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ABSTRACT 

The management information system (MIS) development 
project for California's Regional Occupational Centers and Programs 
(ROC/Ps) was conducted in 3 phases over a 12-month period. Phase I 
involved a literature review and field study to match MIS design 
features and develx)pment strategy with existing conditions in ROC/Ps . 
A decision support system was chosen because of the need for 
integrated and interpreted data reported by local ROC/P managers. A 
middle-out or prototyping approach was selected due to the extreme 
diversity of ROC/Ps. Phase II included development and pilot testing 
of MIS model software at 12 pilot sites. Findings indicated the 
following: the software was relatively free of bugs, technical 
documentation was clear and easy to follow, current data aggregations 
were problematic, data preparation and input time was slow and 
cumbersome, top manager involvement was lower than optimum, and 
conceptual understanding was difficult. In Phase III, participants 
were interviewed and findings were analyzed and reported. 
Participants expressed enthusiastic interest in implementation. The 
most common criticism was the labor-intensive nature of the initial 
data input. Suggested improvements were less restrictive fields and 
changes to user interface. A suggested use for the system was a base 
of program evaluation data. (Appendixes include a 96-item 
bibliography, interview forms, and supporting documents, such as 
lists of data and information needs, review of existing information 
systems, computer model, and software benefits.) (YLB) 



-k-kiz-Jdzric ^'c^V Vc -kic ^'c i< Sc -k -tz -k k -kkk k -k t?c -k z\ tiV k:i\kk'kk-i<kicic Vc z\':'zkkkieickizititkk'kiek i< k -.'c kk-kik k k * k k 

Reproductions supplied by EDRS are the best that can be made * 
''■^ from the original document. 

kkkkkkkkkk'i^ikkkkkkkickk'izkkkkkitkitrkkkierkkkkkiikrkkkkkicickickkickki^^ 



Design of a Model 
f Management Information 
System (MIS) 

for California's 
3 ^^^gio^^al Occupational 
j 00 Centers and Programs 

■ ■ Q 



Final Report 



by 



U.S. DEPARTMENT OF EDUCATION 
Ottice of Educational Rf sexch and impfovement 

EDuCATtONAL RESOURCES INFORMATION 

/ CEMTER(ERtD 
^This document has t>een fep'Oduced as 

*^ 'CCOived from the per^SOn or orgariization 
originating ■( 

C Minor changes have been made to improve 
reproduction quality 

• Points of vie* or opinions Stated '"^ this docu- 
ment do not necessaiiiy represent official 
OERi posittOf* Of cx3iicy 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



7 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



«Fame8 C. Dick 
Douglas E. Mitchell 
Jeffrey B. Hecht 



October, 1991 




CALIFORNIA EDUCATIONAL 
RESEA'::CH COOPERATIVE 



UNIVERSITY OF CALIFORNIA, RIVERSIDE 



EST COPY AVAOBLE 



THE CALIFORNIA EDUCATIONAL RESEARCH COOPERATIVE 
CERC is a unique panne.hip between county and .^^^^^^^^^^^ 

University of California, Riverside. It is designed to scn-e as a J^^^^^^^^^ practical wisdom 

CERC is organi^d to pursue si, bro.d gods. Th»e g.Ms sor,= .ho »«ds .„d interests of cooperating public school 
members and the University by providmg: 



Tangible practical support for school improvement 



• Proven strategies for resolving instructional, 
management, policy and planning issues facing public 
education^ 

• Valuable professional development opportunities 
for current and future school leaders. 



• Support for data-based decision-making 
among school leaders. 

• Research, planning and evaluation activities 
that are meaningfully interpreted and applied 
to school district problems, and 

• Data analysis to assist in generating public 
support for cfTective school programs. 



In addition to conducting research in these areas, CERC publishes reports and briefs on a variety of educational 
issues. CERC also sponsors regional workshops for local educational leaders. 



CAl lFOR\!'\ EDi 'C'\JIO\ -\i 
RESEARCH aX)PER^\T!VE 



CERC 
Executive Staff 



Irving G. Hendrick 

Dean 

School of Education 

Jane L. Zykowski 

Associate Specialist 
Manager 



Douglas E. Mitchell 

Professor of Education 
Director 

3usan Hill 

Administrative Assistant 

Shirley Hall 

Secretary 



Schc«l of Education, University of California. Riverside. CA 92521-0128 (714) 787-3026 FAX: (714) 787-3^)42 



CERC MEMBERS 

SPONSORING OFFICES OF EDUCATION 



Riverside County Office of Education 



Office of the County Superintendent of Schools 
San Bernardino County Office of Education 



SPONSORING SCHOOL DISTRICTS 



ERIC 



Banning Unified School District 
Desert Sands Unified School District 
East San Gabriel Valley ROP 
Fontana Unified School District 
Hemet Unified School District 
Hesperia Unified School District 
Jurupa Unified School District 
Lake Elsinore Unified School District 
Moreno Valley Unified School District 
Murrieta Valley Unified School District 
Ontario-Montclair School District 



Perris School District 

Perris Union High School District 

Redlands Unified School District 

Rialtc Lnified School District 

Riverside Unified School District 

Romoland School District 

Temecula Valley Unified School District 

Val Verde School District 

Victor Elementary School District 

Victor Valley Union High School District 

Yucaipa Joint Unified School District 



Design of a Model Management 
Information System (MIS) 

for California's 
Regional Occupational Centers and 

Programs 

Final Report 



California Educational Research Cooperative 
University of California, Riverside 
1358 Sproul Hall 
Riverside, CA 92521 



James C. Dick 
Douglas E. MitcheU 
Jeffrey B. Hecht 



In contract with the 
California Association of Regional Occupational Centers and Programs 



Page i 



CAROC/P MIS 



Preface 



This is the final report of the research effort conducted by CflVTf>^^?^ 
Educational Research Cooperative in contract with Cplifnmi a Association of 
Regional Occupational O ntprg ^nH Pr^gmmft (nAROryPV This report is 
the major written product of the project* Two other documents which have 
been produced under separate cover are: 

1, Design of a Model Management Information Svstem (MIS) for 
rfllifnmia^fl Regional Occupational Centers and Programs: 
Su pportmijf Docuttients 

This is a collection of 10 supporting documents pertinent to the st but 
not vital enough to be included as appendices in the final report* 

2. Re gional Occupational Centers and Programs - Decision 
Su pport Svstem (ROCP-DSS) Instruction Manual 

This is a manual which accompanies the proto-type software program that 
was developed and tested during the study. It is referenced under T)ick, J. 
(1991)" ir. the BibUography, 

Copies of both of these documents are available through CERC at: 



California Educational Research Cooperative 

School of Education 

University of California @ Riverside 

Sproul Hall 1358 

Riverside, CA 92521 

Phone: (714) 787-3026 
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Executive Siiiumary 



The Problem 

The California Association of Regional Occupational Centers and Programs 
(CAROC/P), one of the largest providers of public vocational education in the state of 
California, has recognized the need for improved information usage in planning, 
developing, managing, and evaluating vocational education programs throughout 
California* Technological advances in the last decade have sharply reduced the cost 
of information management while dramatically improving the reliabiUty and 
sophistication of analysis and interpretation. While making local management 
information systems (MIS) cost effective, however, these new technologies also made 
the job of information ^stem design with the needs and interests of seventy different 
ROC/Ps and the state of California subtle and complex. 

California's Regional Occupational Centers and Programs are sharply divergent 
in size, governance, resources, and local goal priorities. As a result ROC/Ps vary 
significantly from one another in: 

1. specific management problems needing attention 

2. types of information needed to inform management decisions 

3. ciurent capacities to generate and analyze needed information 

4. resources available for MIS development and operation 

5. readiness to incorporate new technologies and utihze advanced 
information analysis techniques. 

Finding a technology and a development strategy to bridge this wide diversity is a 
m^'or challenge to MIS development for CAROC/P. 

In addition to the complexities of local capacity and needs, CAROC/P MIS 
development confronts an inherent tension between uses of information mflnflgpmf^nt 
vs. accountabilitv. To be used for management purposes, an information system must 
identify ways of linking changes in program operation and resource utihzation with 
student outcomes and overall program support. Managers need to understand 
context^ual constraints to be able to explain deviations from expectations and to take 
corrective action. By contrast, v/hen used for accountabiUty purposes an information 
systeni is simpler and more static, Accoimtabihty systenM focus on quantifying 
operations and comparing outcomes. They can ignore contextxial constraints while 
management support systems cannot. Developing a system which balances the needs 
of local managers for context-sensitive details with the needs of state level leaders for 
uniform and accurate summary report information is an essential element in an overall 
CAROC/P - MIS development strategy. 

CAROC/P has had an evolving interest in improved information usage. A 
history of instabiUty in fimding and support has coalesced ROC/Ps into an association 
with the common goal of maintaining their institutional vitality. In the fall of 1988, 
CAROC/P, in cooperation with the state, mvested in a nugor cost/effects study. This 
study recommended that the organization pursue the development of a statewide 
management information system (MIS). 
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The Project 



In August of 1990, the CAROC/P board contracted with the CaUfomia 
Educational Research Cooperative (CERC), a research \mit of the School of Education, 
University of California, Riverside, to undertake development and testing of a 
management information flystem (MIS) for ROC/Ps in Calif omia« A steering 
committee made up of representatives from CAROC/P and the California Department 
of Education set the focus of the project on the design and testing of a model S3rstem 
directly supporting course-level management and also providing a base from which 
state level reports cat. be generated. 

The MIS development project was conducted in three phases over a twelve- 
month period. Phase I involved a review of Uterature and a field study to 
appropriately match MIS design features and development strategy with existing 
conditions in ROC/Ps. Phase 11 included the development and pilot testing of MIS 
model software. Phase HI involved data collection, analysis, and reporting of study 
findings. 

The MIS Piloted 

A prototyping strateg y was selected for MIS development rather than the more 
common hierarchical strategies because of the extreme diversity of ROC/Ps. 
Prototyping begins with the quick creation of a small, workable, modular system which 
is focussed on a particular problem or set of problems. It is implemented next to 
existing sjrstems. Users provide feedback to the designers on the fit of the system 
features to their situation. Designers incorporate changes into subsequent versions 
of the model which are further tested and critiqued. This iterative cycle continues 
with refinements and expansions of the system as long as the development is 
beneficial to the users and/or cost effective for the supporting agency. In this case the 
prototyping approach allowed the system to focus on the management problems and 
decisions common to all ROC/Ps and avoided the conflicts and costs associated with 
constructing a system at the level of daily operations. Requiring uniformity of 
operations at the local level was suspended in the short term until consensus could be 
reached on the management decision priorities. 

A decision modeling desig n was taken rather than a standardi", d data analysis 
design because of the need for flexible data analysis in local ROC/i -s. The lack of 
clarity on the priorities of different program quality indicators in ROC/Ps called for 
a system which could model the relative importance of various measures. By allowing 
local managers to model data anal3rsis after their own imique decision mal ^g patterns 
and priorities, the system becomes a tool for shaping and documenting dedsions. For 
the prototype MIS development, standardization of data analysis was de-emphasized 
in favor of flexibihty and adaptabiUty to local differences. 

A Decision Support System (DSS) rather than a operations control system was 
developed becaxise of the need for integrated and interpreted data reporting by local 
ROC/P managers. The type of DSS most appropriate to the conditions in ROC/Ps is 
an analysis in fnrmfl^iQn system which integrates and analyzes data on costs. 
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enrollments, retention, attendance, ADA revenues, labor market, completions, 
placements, quality of instruction, student and employer feedback, and other measures 
of coia^ quality. The piloted system fiadlitates operational management and strategic 
planning related to ROC/P programs. It also helps ROC/P management to quickly 
identify programs needing attention, target interventions, and provide documentation 
to demonstrate program quahty and justify management decisions. The DSS 
developed for CAROC/P serves as a tool to improve local management and increase 
local motivation to collect valid and reliable data. In its piloted format, the system de- 
emphasizes external control through standardized reporting of quality indicators. 
When local management has achieved a level of confidence in using information to 
support decisions and when consensus has been reached on conunon indicators of 
merit, the establishment of ancillary reporting functions to develop statewide analysis 
of overall ROC/P productivity will become feasible and valuable. In the long run, the 
capacity of the prototype MIS to share and report common information will be virtually 
unlimited. 

Conclusions 

Uses of the System 



The system has four m£Oor functions and ten primary benefits as illiistrated in 
the following table: 



Functions 


Benefits 


A. Information Organization 


1. Integrates data from many sources 


2. Sorts available data based on multiple 
weighted variables 


B. Information Analysis 


3. Refines data definitions 


4. Establishes local standards 


5. Clarifies priorities 


C. Decision Support 


6. Localizes management control 


7. Monitors accounting and reporting 


D. Comnivmication 


8. Informs of conditions 


9. Motivates to action 


10. Justifies decisions 



The MIS Improves Information Organization 



Many ROC/Ps have separate information systems for finance, attendance, 
personnel, courses, and students. The model MIS does not replace these, but it 
integrates data from each into a single database describing the characteristics and 
operations of all sections operated by the ROC/P. By establishing the course-section 
as the universal vmit of analysis, the prototype MIS makes a wide variety of analyses 
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possible. The system sorts sections in ranked order to show relative performance on 
any operational characteristics of interest to local managers. Managers need only to 
glance at reports to see which sections are performing at or above expectations and 
which are not. Another organizational feature of tiie prototype system is tiiat it 
reports results using uncompUcated plus (+ ), minus (-), and neuti^ (o) symbols rather 
than overloadmg reports with numbers. Reports show quickly which aspects of section 
performance need attention. 



The MIS Enhan ces Data An»1v«i« 

Important universal performance measures, such as labor market and 
instructional quality, can be derived from combinations of several indicators The 
prototype MIS helps ROC/P managers refine Uieir data definitions by merging 
multiple measures. 

ROC/Ps are required by law to establish quality standards for tiieir courses 
This function is facilitated in the model by allowing users to set two break points for 
each measure or detail defined in the system These break points estabUsh a positive 
a neutral, and a negative range. Since these are locally determined, tiiey can be used 
for local goal setting and motivational purposes. 

The developed system recognizes that program performance can only be 
adequately determmed when multiple indicators are used. Management needs 
pnonties and options are brought into focus by generating course performance profiles 
based on multiple indicators and locally-determined success criteria. The prototype 
MIS has a system for weighting details so the relative importance of different 
mdicators can be clarified through "what if modeling 



The MIS Sup ports Improved Decisio n MaVing 

Like other educational and social service agencies, ROC/Ps require 
sophisticated management information systems in order to significantly improve 
management decisions. Only by putting a powerful management tool into the hands 
ot local managers can information analysis be helpful enough to offset data 
management costs. Properly used, the prototype MIS provides feedback to managers 
to help Uiem gmde both mterventions and commendations. It gives clear indication 
of where tiie attention of the manager is needed and how serious any given problem 
might be, rektive to other problems. It shows when goals and objectives have been 
achieved, and when sections or courses need to be terminated. 

FutureyersionsoftheprototypeMISwillbeequippedwithareportingmodule 
making It possible to share data between ROC/Ps and report summary data directly 
to the state Local managers will be able to monitor accounting and reporting since 
the mformation reported will have akeady been used to support basic Lnagement 
functions The model system may eventually replace the current processesfbTcourse 
approval, biennial course quality review, and compliance reviews. 
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The MIS Improves Cn TnTnnnii^^ ^ion with Kev Constituencies 



Both internal and external communication in ROC/Ps is limited by a lack of 
integration and interpretation of data. The prototype MIS provides flexible control of 
data reporting, and uses a simplified format which is easy to understand. These 
reports have value at all levels of the organization to describe cxirrent conditions and 
motivate action to improve program operations. Additionally the reports benefit 
managers by justifying the decisions they make, especially those in sensitive areas. 
The reports are a powerful tool for communicating botii the strengths and the 
weaknesses of the organization. 



Limitations of the Pilot Version 

Some limitations of the current test version of the program are obvious. The 
most conspicuous is that all data must be collected and transformed outside the 
program itself. At this point, the system is designed exclusively as an engine for data 
analysis and reporting. Data collectio a and storage fimctions remain with each ROC /P 
to manage us:ng procedures used in the past. A production version of the software 
would have a much more sophisticated electronic interface with existing data systems 
which would automatically import data and transform it appropriately. 

Need for Manager Development and Technical Support 

The level of manager involvement and conceptu^ understanding in the first 
round of model testing was lower than needed for long term implementation of a fully 
developed MIS. Many ROC/P managers assumed that the prototype system was 
intended to improve data processing efficiency rather than support management 
decision making activities. Since the DSS model was not designed as a transaction 
processing tool to improve clerical efficiency, those who treated it as such were 
disappointed and usually did not imderstand the system's power as a management tool. 
Clearly a mfgor hurdle to a broad-based understanding and acceptance of the DSS 
model for ROC/Ps is an education of managers to the fundamental differences 
between transaction processing systems and management information systems. 



ERIC 



Page vii 

10 



CAROC/P MIS 



Recommendations 



Further System Development 

Among the first additions to subsequent versions of the DSS software must be 
modules that allow menu-driven user control over electronic transfer of data into the 
system* The overhead costs of data entry must be reduced significantly if the DSS is 
to be regularly used by ROC/P managers. A second priority in^ DSS revision is the 
added capabUity of calculatmg new details from existing details and of automatically 
aggregating and disaggregating data in various ways. A third priority addition is the 
capacity of the DSS to store historical data and perform trend analyses and reporting. 
Other suggested additions to a second version include: 

1. capabihty to suggest solutions to problems found 

2. opportunity to interactively insert memos into reports 

3. more section and course code fields on which to select 

4. more cut-off points in details for greater case discrimination 

5. a module for generating the state required reports 

Technical improvements suggested include: 

1. allow changes of the colors for monochrome screens 

2. use function keys to shortcut some of the processes 

3. allow for customized data viewing and reporting 

4. provide more on-line help 



Implementation Support 

Full implementation of a production version of the MIS model will need 
substantial support to be a success. First of all^ the association should pursue a second 
round of pilot testing with the added improvements as listed in an above section. 
Along '^th the testing of ^stem improvements a substantial staff development effort 
is needed to train ROC/P managers in information-based man^ement decision 
making. Included in such an effort would be the development of promotional and 
training materials, on-site management consulting, and support for regular training 
sessions. 

Following the pilot testing of a second version of the ROCP-DSS, and the 
preliminary training of ROC/P managers, a production version of the DSS should be 
developed for distribution and implementation in all ROC/Ps statewide. The 
implementation process will require training support in the use of the ^stem for 
management improvement. In-house technical consultants wiU also be needed to assist 
in installing the system and in creating customized local linkages to existing 
transaction processing systems. 
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To realize the full potential of the S3^tem for networking and sharing 
information statewide, resoiirces must be made available to facilitate (a) the sharing 
of decision models and (b) the standardization of data element definitions and data 
collection protocols. Once the consensus has been built at the local level as to the 
most appropriate types of data to distribute pubUcly, the electronic system for linking 
all ROC/Ps to one another and to the state office of education can be designed and put 
into place. 



State Policy Framework and Support 

To state poUcy leaders we offer the following recommendations: 

1. Take the long view of developing local capacity for improved information-based 
ROC/P management rather than the short view of developing accuracy in data 
reporting. 

2. Encourage and faciUtate local consensus building on data definition and 
collection protocols. Make it attractive for local ROC/P managers to use 
information for management purposes. 

3. Create a state level capacity for re-analysis of local models to learn how local 
managers model their own decisions and how local context variations catise 
managers in apparently similar decisions to make different decisions. 

4. Examine state policy and regulations for ways to increase the accommodation 
to local variations. Allow local managers to adjust programs to local needs so 
that local information will become more valuabl- Valued information is much 
more likely to be vaUd and reliable information. 
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Chapter One 




Introduction 



This project. Design of a Model Management Information System (MIS) 
for California's Regional Occupational Centers and Programs, had five goals: 

1. Determine the nature of the MIS needed in ROC/Ps 

2. EstabUsh an MIS development strategy 

3. Design a model system 

4. Pilot test the model in several sites 

5. Evaluate the model and report on its utility 

Particular attention was given to addressing the wide variation among ROC/P 
managers in identifying and defining course level management information 
needs. The expected outcome of the project was a model MIS c;ipable of 
anal5rzing ROC/P programs on such variables as student enrollment, program 
quaUty indicators, labor market demand, and program outcomes like placement 
rates. Specific design elements were developed through involvement and input 
of state level staff, ROC/P site managers, and experienced researchers. 
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Background and Setting for the Study 

For over a quarter of a century Regional Occupational Centers and 

Programs (ROC/Ps) have been the chief providers of state funded job training 

for high school jtmiors and seniors, ROC/Ps also provide adults with original 

job training, skills upgrading, and/or retraining. The unique mission of 

ROC/Ps is to provide vocational and technical training to prepare students for: 

an increasingly technological society in which generalized training and 
skills are insufficient to prepare high school students and graduates, and 
out-of-school youth and adults for the many emplo3rment opportimities 
which require special or technical training and skills. (State of 
California, Cal Ed Code, Section 52300) 

During the decade of the 1980's several factors contributed to the need 
for a management information system in California's ROC/Ps. Fluctuations in 
the state funding allocated to ROC/Ps, caps placed on revenues, increased 
accountabihty reporting requirements, and changed high school graduation 
requirements all impacted ROC/P management (Mitchell & Hecht, 1989). 
These pohcy changes, increased corporate and pubhc interest in vocational 
training dramatically and increased the complexity of the task of managing 
ROC/Ps. By the late 1980s ROC/Ps were serving fewer high school students, 
attracting more adults, and modifying their training programs to contain costs 
in a rapidly changing high-cost workplace. 

The California Department of Education and the California Association 
of Regional Occupational Centers and Programs (CAROC/P) have been 
cooperatively seeking to clarify the management needs and develop meaningful 
solutions. In 1987 a research project was funded by the state for CAROC/P to 
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evaluate and demonstrate the cost effectiveness of ROC/P programs. The 
findings from this study, conducted by the Caiifomia Educational Research 
Cooperative (CE"^C) confirmed that ROC/P revenues have fluctuated 
dramatically and have gradually decreased over the past 20 years. The study 
also demonstrated that the program outcome data intended to document 
ROC/P performance are neither common, useful, nor reliable (Mitchell & 
Hecht, 1989). 

Without commonly defined and reliably collected data, any attempt at 
the analysis of program effectiveness is subject to error and open to criticism. 
Statewide data are needed for making sound poUcy decisions. Locally specific 
program quality measures are needed for making inteUigent program 
improvement decisions. Existing information systems are unable to provide 
useable data for either state or local purposes. As a result, the ROC/P cost 
effects study report recommended that the state and CAROC/P collaborate on 
the development of a Management Information System which could address the 
needs for accurate and reliable data at both the local and the state levels 
(MitcheU & Hecht, 1989) 

Scope of the Project Activities 

In 1990, the California Department of Education (CDE) funded this 
project to research, design, and test a model Management Information Svstem 
(MIS) for ROC/Ps. The Caiifomia Association of ROC/Ps again subcontracted 
with CERC to research and develop a model management information system 
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which could be implemented on a state*wide basis. The research was 
facilitated by an advisory committee formed to provide direction and guidance. 

The study was undertaken in three phases over the twelve month period 
between September 1, 1990, and August 31, 1991. In Phase I a review of the 
literature on information system development was undertaken to identify 
possible approaches to take. A field study was then conducted to determine the 
nature of needs and conditions in ROC/Ps which would help to direct the 
design and development process. In Phase II a model MIS with supporting 
documentation was developed and pilot tested in twelve selected sites across 
the state. In Phase III participants in the pilot study were interviewed end 
findings were analyzed and reported. Recommendations for further 
development and implementation are presented in the Executive Simmiary 
preceding this report. 
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- Phase I - A Literature Review on MIS - 

A substantial body of theoretical and empirical literature on Management 
Information Systems has been produced within the last twenty years. With the 
rapid growth of the computer industry, the study of automated information 
systems has expanded in a number of directions. Information system analysis 
has found its way into such diverse fields of study as sociology, electrical 
engineering, cognitive psychology, and organizational behavior (Ahituv & 
Neumann, 1990). This body of Uterature was reviewed to provide an 
understanding of the definition of management information systems and to 
identify the optional approaches to information system development. 

Appropriate matching of information system design features and 
development strategies to the situation where the system is to be implemented 
requires a thorough unde- standing of aU three - design features, development 
strategies, and situation conditions. The Uterature review in this chapter 
clarifies optional design features and development strategies, the field research 
described in chapter three identifies the prevailing conditions in ROC/Ps which 
pertain to the selection of design features and development approaches. 
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Definition and Components of an MIS 

Management information systems, and the acronjon MIS, have been 

given many different meanings. One of the more comprehensive definitions is 

provided by Murdick (1980, p. 11): 

The system which monitors and retrieves data from the environment, 
which captures data from transactions and operations within the firm, 
and which filters, organizes, and selects data and presents them as 
information to managers is called the management information ^stem 

It is important to distinguish between Transaction Processing Systems (TPS)- 

and Management Information Systems (MIS). TPSs capture, store, and report 

data MISs go further, they organize and transform data into information; for 

generating management reports in the form of: 

1. summary periodic reports to monitor organizational performance 

2- operational exception reports to highlight potential problems or 
identify new opportunities 

3. strategic planning and control reports to analyze decision options 

(Murdick, 1980). 

High level management information systems are sometimes called 
Decision Support Systems (DSS) (Alter, 1976). This label highlights the 
managerial role of the system and focusses attention on the difference between 
DSS and lower level Transaction Processing Systems (TPS) which are primarily 
concerned with the techniques and procedures for data storage and retrieval. 
The differences between transaction processing systems and decision support 
systems are described in Figure 2.1. 
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Comparison of purposes, uses, and characteristics of 
Transaction Processing Systems (TPS) 
and Decision Support Systems (DSS) 



Chur»ct«r- 
Utios 



TranMcUon prooeMlng systems 



Transaction processing 
Record keeping 
Business reporting 



Obtain pre-spedfied aggregations of 
data in forms of standard reports 



Passive clerical activities; 
Oriented toward mechanical 
efficiency; Focus on past; 
Emphasis on consistency 



Decision support systems 



Decision making 
Decision implementation 



Retrieve isolated data items; Use as 
mechanism for ad hoc analysis of daU 
files; Obtain pre-spedfied aggregations of 
L.''^ in the forms of standard reports; 
Estimate consequences of proposed 
decisions; Propose and Make decisions 



Active line, staff, and management 
activities; Oriented toward overall 
effectiveness; Focus on present and 
future; Emphasis on flexibility and ad hoc 
utilization 



(adapted from Alter, 1976:98) 



Figure 2.1 
Comparison of TPS and DSS 

The key operational terms for a TPS are efficiency, reUabiUty, and 
consistency. To be useful the TPS must keep both hardware and staff uses to 
a minimum and still move data rapidly and reUably from input to report 
formats. The criteria for evaluating a DSS is quite different. To be valuable, 
a DSS must b p ca pable of combining da t a creatively and flexibly and extracting 
o perational parameters that improve m anagement control systems. 

Some analysts distinguish communication support systems from decision 
support and transaction processing systems as different elements of a 
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comprehensive information management system (Kozar, 1989) Figure 2,2 
outlines the contents of these three different types of information systems. 



Types of information managemsnt systems 



Operational Support 



Transaction ProcMsing 
Scheduled Reportins 



DecMonMaidno Support 
systims 



inquiry 
-Ad hoc report* 

Analysis 



Conmunlcatlona Support 
systems 



Text Handling 



-Qra^Mco 

-DooiBfMnt producdoci 
Teieoommunications 

-OoniirtnelnQ, Mai» vote 
Ring 
- Ci i tnd i f i, Inctot 
'Mdo llfn^/Hche 

(KDanr,1S6a:14) 



Figure 2.2 
Types of Information Systems 



Before the technology explosion of the 1980s, virtually all efforts at MIS 
development concentrated on transaction processing. This emphasis was made 
necessary by the high cost of electronic data processing hardware and the 
limited availability of trained staff capable of operating it efficiently. Reporting 
of data of necessity, emt hasized development of list oriented transaction 
records. These systems, usually designed around several subsystem data bases, 
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are expensive to integrate and generally take a long time to design and install 
properly. Kozar (1989) notes an increasing shift toward the development of 
decision making support and communications systems in recent years. These 
more recent systems are designed to: 

- meet immediate needs of managers for information 

- respond to time pressures to solve problems 

- focus on specific problems (often short Hved) 

- require some pre-anaiysis or manipulation of data 

- incorporate data from a variety of sources 

Decision making support systems tend to be less elaborate in design than 
transaction processing systems. Involvement of organization members in 
identifying the goals of the system and the immedia^ of the data analysis 
problem precludes long software and data collection «^^ve opment times. It is 
often useful to qmckly develop prototypes when desigiivag decision making 
support systems (Kozai% 1989). 

As noted by Ahituv and Neumann (1990) there is a substantial overlap 
between decision support and transaction processing system functions. They 
label the data storage e^d retrieval as well as the operations control associated 
with automated data manipulations as administrative data processing* Figure 
2.3 illustrates how these authors reserve the term MIS for the systems that 
include information analysis and automated control of structui?d decisions. 
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A Physical Structure of 
Information Systems 
in Organizations 



Organizational 



Decision 
Support 
System 



Management 

Information 

System 



Information 



System 



Administrative 
Data 

Processing 
S5rstem 



Structured 
Decision 
System 



Transaction 
Processing 
System 



(Ahituv & Neumann, 1990:131) 



Figure 2,3 

Structure of Systems in Organizations 



They conclude by defining an MIS as: 

an information system that makes some managerial decisions and 
provides managers at all levels of an organization with the information 
needed for making other decisions (Ahituv & Neumann, 1990, p. 133). 

Relation of Decision Support to Management 

Managers differ in their needs for information. This difference is related 
to the types of decisions made at different levels in an organization. Top level 
managers must solve problems which have little structure and for which the 
data needs are poorly defined as illustrated in Figure 2.4. 
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Management Hierarchy and Information Uses 



PaoilydifiMd 
orundaAnwl 



Unabucbnd 
preUxns 




WMI 

datinMdt 



Operational Personnel 
Operational Tasics 



Structur»d 
prafaianw 



Figure 2.4 

Management Hierarchy and Information Uses 



Managerial decisions can be classified into the three levels: 

1. Strategic planning - Deciding on objectives of the organization, on 
changes, on the resources needed, and on the poHcies that are to govern 
the acquisition, use, and disposition of these resources. 

2. Management control -- Assuring that resources are obtained and used 
effectively and efficiently to accomplish the organization's objectives. 

3. Operational control -- Assiuring that specific tasks are carried out 
effectively and efficiently (Ahituv & Neumann, 1990, pp. 111-112). 

Another way of representing information needs at different levels of 
management is by information types. Top managers have a relatively low need 
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for detailed factual reports on the internal working of the organization. They 
have a much higher need for sununarized status reports on the relative health 
of the organization as well as external intelligence related to the impact of the 
organization on its constituents. Figure 2.5. illustrates the relative need for 
different information types at different management levels. 



Management information Needs Differ by Level in the Organizatk>n 




Topmanagm 
andcxKUtivw 



OpmtinQand 
managm 



Proportion of Infomiation types needed 



Oomioit h&wnision 
min» iinMiiiiion 
WMiring intoniwlbn 

I'fWWiQ nninrauon 

EoImtmI op9fttiOM IrtfocTiMtkin 

BdbmailnMliOHiM 

E)flwTiily diittfcutKl jntonnitton 



Factual detalto 
Excaption raport» 
RnandflJ accounting Infomiation 
jytonagament accounting infonnadon 
Intimaily orianttd tnfennatSon 



Figure 2-5 
Management Information Needs 

Types of Decision Support Systems 

Different types of decision support ^sterns can be designed to meet 
different management information needs. The type of system depends on the 
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nature of decisions being made, the nature of the organization, and available 
information processing capabilities. Alter (1977) examined a large number of 
decision support systems and found significant design and functional 
differences. He advanced a taxonomy of decision support systems which 
distinguished systems on several key characteristics. Figure 2.6 shows seven 
types of systems by orientation, action, type and function. 



A Taxonomy of Decision Support Systems 



Orientation Action Type Function 



DaU 


Data 
Retrieval 


File Drawer 
Systems 


allow immediate access to data items. 


Oriented 


Data 
Analysis 


Data Analysis 
Systems 


allow the manipulation of data by 
means of operators tailored to the 
task and setting or operators of a 
general nature. 






Analysis 

Information 

Systems 


provide access to a series of data 
bases and small modules with ability 
to integrate and analyze data. 


Model 


Simulation 


Accounting 
Models 


calculate the consequences of planned 
actions based on accounting 
definitions. 


Oriented 




Representational 
Models 


estimate the consequences of actions 
based on models which are partially 
non-deflnitionaL 




Sviggestion 


Optimization 
Models 


provide guidelines for actions by 
generating the optimal solutions 
consistent with a series of 
constraints. 






Suggestion 
Models 


perform mechanical work leading to 
a specific su^ested decision for a 
fairly structtired task. 



(Alter, 1977:41-42) 



Figure 2.6 

A Taxonomy <Df Decision Support Systems 
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A file drawer system provides on-line access to an existing electronic data 
base. In an ROC/P setting such as system would allow a manager to access up- 
to-date reports like the enrollment status of any given class. Such a system 
keeps administrators efficiently informed with current information pertinent 
to organizational operations. 

A data analysis system is slightly more complex in that it screens data 
before reporting it. Summaries, graphical representation of numbers, and 
sorted lists are typical outputs of data analysis systems. This type of system 
also keeps the manager informed of operational conditions, especially problems 
and exceptions. 

Analyaig i nformation systems are designed to integrate, screen, and 
interpret information. They extract relevant information from existing 
electronic data processing systems, combine it with information from other 
internal or external sources, and perform various sorts, calculations, or analyses 
on the data before reportmg it. They are often rather flexible in that managars 
can select the information to include in the analysis and can manipulate the 
reporting to match their changing information needs. In ROC/Ps an analysis 
information system might combine data relative to course costs, completions, 
placements, and job market to produce a profile of course quaUty. Such 
reportmg capabilities are useftd to top level managers for strategic planning and 
for middle managers and program supervisors for appropriately targeting their 
inservice and intervention efforts. 
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Accounting models function primarily as planning tools to simulate or 
project the consequences of a particular decision or action. These models work 
particularly well with budget planning and other cost analysis management 
decisions which lend themselves to clearly defined data relationships and 
specific numerical formulas. They are usefiil to business managers for planning 
budgets and analyzing the cost benefits of certain decisions. 

Representational models function similarly to accounting models in 
simulating the outcomes of a particular action. These models differ firom 
accounting models in that the analysis is based, at least partially, on 
relationships which are not clearly defined- Representational models often 
have a secondary function of helping to clarify the natinre of the relationship 
between various forces influencing decisions or outcomes. Such a model appUed 
to ROC/Ps may be used to clarify the relationship of various quality indicators 
such as labor market, cost, or placements to the decision of the ROC/P director 
on course retention or termination. 

Optimization models are based on complex mathematical treatments of 
data which have as an end output recommendations for how to best reach goals 
such as maximizing income or minimizing cost. Optimization models are often 
seen as a "way of viewing tradeoffs, the importance of constraints, and so on." 
(Alter, 1977, p. 48). 

Sug gestion models are much more structured than optimization models. 
They function to perform complicated calculations which lead to the best or the 
right answer to a problem situation. The output of such models is a **specific 
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recommendation for action" (Alter, 1977, p. 49). They serve to supplement or 
replace some of the routine but complex thinking and calculations which a 
manager would otherwise have to do in making a decision. Another name for 
a suggestion model is an expert ^stem. 

Strategies for Information System Development 

Just as there are many different types of management information 
systems, there are multiple strategies for designing and developing information 
systems. A careful analysis of development strategies suggested m the 
Uterature provides a sound basis for approaching MIS development for 
CAROC/P. At least four different appropriate strategies to system 
development can be identified (Ahituv and Neumann, 1990). They are: (1) 
bottom-up, (2) evolutionary or modular, (2) top-down, and (4) middle-out or 
proto-typing. These approaches can be illustrated in relation to the order in 
which they place emphasis on the different components of a system. 

Information System Components . 

The goal of system development is a complete and comprehensive 
information system including transaction processing, structured decision control 
and unstructured decision support systems. Figure 2.7 illustrates these major 
components of a system. The transaction processing system (TPS) supports 
operational staff and emphasizes data processing efficiency. The structured 
decision system (SDS) takes information from the TPS, controls routine 
decisions and generates standard reports needed by managers. A decision 
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support system (DSS) provides executive management with information needed 
to solve planning or policy problems. A DSS assists with decisions that are ill- 
structured or require a broad range of information and inputs. 



Information System Components 

MIS M«n«o«m«fttlnfbrmationSy«iMi 

CIS 



Figure 2.7 
Information System Components 

Bottom-up Development 

A bottom-up strategy of information system development is based on the 
premise that data must first be captured, confined, defined, and organized 
before it can be turned into information that is useful for management. 
Transaction processing is the primary focus of bottom-up development 
strategies as illustrated in Figure 2.8. Proponents of this approach suggest that 
the TPS is the tool through which the SDS and DSS are made to work. They 
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argue that until data storage 



and retrieval are fully in place 



The bottom-up approach 



pursued. When using this 



expended on understanding 



strategy much less effort is 



the more advanced 



applications can not be 



Dcui dtfbiltlon 
andooledkn 





the global framework of the 



Figure 3.8 



information system than on the more descriptive analysis of the flow of 
operational information within the organization. The bottom-up approach is 
driven primarily by the efficiency of data processing. Most textbooks on 
information system design describe some form of this approach, (see Doukidis, 
Land, & Miller, 1989; Conger, Colter, and Knapp, 1982; Kozar, 1989; Matthews, 
1981; Murdick, 1980) By virtue of the historical growth of electronic data 
processing technology from simple data storage and retrieval technologies to 
complex analysis programming the bottom-up approach paralleling this 
historical development is the most common in reality (Hopple, 1988). 

An example of the bottom-up development approach within California's 
ROC/Ps is the system developed for San Diego County ROP. According to a 
consultant from the computing corporation with which the ROP contracted, the 
firm responded to the needs of the ROP to improve the efficiency of its 
operations. Over a period of four or five years the firm buHt a data system 
which significantly automated the transaction processing functions. Only in 
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recent years have several important structiured decision control and 
unstructured decision support elements been added to the systenL 

The bottom-up approach is most appropriate when efficient data 
processing and uniform definition of data elements are the high priority 
outcomes of the ^stem. A bottom-up design strategy requires a high 
conmiitment on the part of all involved to the standardization of data and data 
manipulation processes. This is usually achieved when the data definitions and 
data uses are already clearly established. 

The disadvantages of the bottom-up approach are that the creation of 
well a structured data base can often become an end in itself and can distract 
fi-om a decision maker perspective. Management apphcations often appear as 
after-thoughts which are tacked on to a TPS. By failing to estimate 
management information requirements in advance, designers often fail to 
optimize integration in later stages of system development. Also as the 
management needs grow and change the TPS may have to be redesigned to 
accommodate changes which were not adequately anticipated in the beginning. 
Another problem which is particularly evident in the bottom-up approach is 
that top managers are often only indirectly affected by initial stages of system 
development and are thus not integrally involved in the planning and 
development. When the time for DSS development arrives managers who have 
been uninvolved in prior developments may have a tendency to defer to the 
system designers for a DSS which can be designed within the limited 
parameters of the TPS. 
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Evolutionary Development 

The evolutionary 
approach, also known as the 
modular approach to infor- 
mation system development is 
based on the premise that 
organizations and their 
information needs are in a 
constant state of flux. Rather Figure 3.9 

than attempting to design and develop a comprehensive, integrated system, 
those who choose this approach build only specific subsystems on an as-needed 
basis. These subsystems are often self-contained, and are only integrated with 
other subsystems when necessary as illustrated in Figure 2.9. An example of 
this approach in ROC/Ps is the development of the ^Socrates" attendance 
system. This system was developed to meet a specific need in ROC/Ps. It was 
designed to fimction as both a TPS and in some ways as an SDS. It was not, 
however, integrated with financial information subsystems. Further details are 
provided in Supporting DnciiTnpnt #8 (see preface). 

The advantages of the evolutionary approach is that it is one of the least 
intrusive of the approaches to system development in that it meets an 
inmiediate need and is focussed on a specific type of information. Because the 
modules of such a system tend to be smaller and independent, the system as 
a whole may be better able to adapt to changes in the information needs of the 



The evolutlonaiy approach 
a.k.a. the modular approach 
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organization. Given the fluctuations in funding and the changes in reporting 
requirements for ROC/Ps in the last ten to fifteen years (Mitchell & Hecht, 
1989) it is not surprising that most ROC/Ps have adopted more of an 
evolutionary or modular approach to their information system development. 

The primary disadvantage of the evolutionary approach is that system 
integration is particularly difficult at later stages of development - just when 
such integration becomes most important. The lack of integration between 
subsystem modules precludes support for complex managerial decisions. Ahituv 
and Neumann (1990) also suggest that the evolutionary approach is an extreme 
form of the bottom-up approach and thus shares many of its disadvantages. 



ERIC 



Top-down Development 
The top-down devel- 
opment strategy illustrated in 
Figure 2.10 is based on the 
premise that the system exists 
to serve the needs of the 
management. It presumes 
that the system development 



Th© top-down approach 
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Figure 3.10 

can not begin before the complete management needs for data have been fiilly 
described While the actual physical development of the system may not begin 
with the decision support, these applications are purposefully integrated into 
the original system design. 
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The top-down approach is most appropriate when a great deal of 
importance is attached to the use of information for management. It works 
well when the various data sources are already well integrated. It generally 
requires an extended period of time for studying and documenting management 
information needs prior to development. 

Systems developed with the top-down approach sometimes die of their 
own weight as more and more data needs are added to the system design 
before any development actually takes place. An example of this problem is the 
information system developed for the community colleges in the state of 
California (Hamre & Holsclaw, 1989). The system was designed to be 
comprehensive enough to meet everyone's management information needs, but 
implementation has been particularly difficult because the system is so huge 
and cumbersome. In the real world, the immediate needs for detailed 
information and efficiency of processing often preclude a very thorough-going 
top-down approach to system design. 

Middle-out Development 

The middle-out, or proto-typing approach, is a new strategy made 
possible by recent technology developments. As hardware and software have 
increased in sophistication, the time required to program computers for 
complex analysis functions has been significantly shortened. Routines which 
only a few years ago took months of programmer time to develop are now 
available by selection from a menu in generic software packages (Fisher, 1991). 
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These changes make it 
possible to design proto-typicai 
information systems and 
subsystems relatively quickly. 

The proto-typing 
process involves piloting a 
system, providing feedback to 
the system designers, 
modifying the system for a 



The mlddld-out approsd) 
alca. tha piiMoiypIng approach 





Figure 3.11 



better fit, and trying it again. This approach is similar to the evolutionary 
approach but it tends to begin with the development of decision oriented 
modules rather than with improved transaction processing. It often presumes 
the existence of at least some sort of TPS upon which it builds, but may even 
start in an environment which is not at all automated. The focus of the 
middle-out approach is on solving a key management problem central to the 
organization's success. Once the targeted problem is adequately solved by the 
proto-type system, the system can be expanded to encompass other 
management areas or another module can be added as illustrated in Figure 
2.11. This strategy is particidarly useful in environments which are unstable 
or where the natiire of the information needed is not immediately clear. 

There are several advantages to the proto-typing approach. First of all 
it makes information inunediately useable for management decision making and 
thus has a much higher chance of being accepted and promoted by top 
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managers. Since top management acceptance and involvement are key factors 
in information system success (Alter, 1976; Hill, 1987) this characteristic may 
be particularly valuable in situations where resistance is likely or expected 
Proto-typing not only quickly solves a real problem, but it "pro^'ides feedback 
on the structuring of the problem and on the use of the technique" (Ahituv & 
Neu*-^ jm, 1990, p. 246). Proto-typing is an inexpensive way to explore both 
the nature of a problem and the possible design of a systematic solution. This 
is particulaHy useful in situations where it is not clear whether to invest 
heavily ic em development. Proto-typing, with its modular nature, also has 
th" -ivantage of being able to tie into existing TPS systems. This feature is 
w-^vantageous in situations where many different TPSs are already in place. A 
prototype can be designed to test the possibihty of integrating data from 
several non-integrated TPSs. The modular nature of the proto-typing approach 
gives it the advantages of the evolutionary approach in being more responsive 
to changes in data needs. 

The disadvantage of the proto-typing approach to information system 
development is that it does not directly address transaction processing issues. 
It either presumes an existing TPS or expects management enthusiasm for the 
value of the DSS to drive the development of data collection and transaction 
processing systems. Proto-typing may not be the most efficient in a stable 
environment where the natm-e of the problem is clear and the information 
needs for both operation and management are already well defined, and where 
few changes in information needs are expected. 
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Summary and implications of literature review 

The primary objective of this project was to design an appropriate model 
MIS for California's ROC/Ps. This literature review clarified the optional 
design features and developmental strategies and raised the following questions 
to be answered by the field study. 

1. Do ROC/Ps need primarily a 

a. transaction processing system (TPS), 

b. structured decision control system (SDS), or 

c. decision support system (DSS)? 

2. If a DSS is needed by ROC/Ps, what types of decisions most need 
support? Which type of DSS would best support these decisions? 

3. Which development strategy would be most appropriate for the 
type of system needed and the prevailing conditions ROC/Ps 
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Chapter Three 3 




- Field Study to Determine MIS Needs - 



Phase I of this project was largely exploratory. The findings of the 
previous research clearly highlighted the need for a management information 
system for California's ROC/Ps, (Mitchell & Hecht, 1989). The literature 
review had identified different types of information systems and several 
optional development strategies available. Further field analysis of the 
dynamics of the situation in ROC/Ps was necessary to determine both the t3Tpe 
of ^stem needed and the most appropriate development strategy to pursue. 
This chapter describes this field study of conditions in ROC/Ps. 

The advisory committee, made up of representatives firom the state 
department of education, the CAROC/P association, local ROC/Ps, and the 
research team firom the University of California, met early in the project to 
define the project parameters. In the first advisory meeting, held on August 
2, 1990, the committee discussed the goals, defined the scope and clarified the 
foc\is of the study, and identified the appropriate sources of feedback for the 
exploratory phase. Table 3.1 outlines these parameters of the field study phase 
of the project. 
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Table 3.1 

Parameters of Phase I of the MIS Project 



PARAMETER 


DESCRIPTION 


Goals 


Clarify the nature of the problem and identify 
characteristics of an appropriate MIS model. 


Focus 


Information needs for effective local management. 


Scope 


Limited to the life cycle of a course. 


Methods 


Naturalistic qualitative analysis: discussion, 
brainstorming, unstructured interviews, observation. 


Sources of 

Feedback 

Information 


Advisory committee, ROC/P managers, special interest 
gioups, MIS software currently in use, and site visits 
in representative ROC/Ps. 



In discussing the primary goals of the project in the advisory committee 
it became apparent that the needs and interests of the groups represented 
were somewhat different. The representatives from the state department of 
education were interested in answers to the questions: 

1. Vvho is being served? 

2. What programs are offered? 

3. What are the expected outcomes and accomplishments? 

4. What are the associated costs? 

5. What are the sources of income? 

These questions impUed a reporting system which would captiure largely 
descriptive data. 

Representatives from the association of ROC/Ps defined the problem 
much more in terms of the needs for data at the local and association level. 
Their descriptions of the ideal system included not only a comprehensive data 
collection and storage system at each local site, but also an electronic network 
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linking sites with each other. The system should allow local managers to 
identify the strengths and weaknesses of their programs and share selected 
information among other ROC/Ps* Individual ROC/Ps could benefit from 
shared information, and the association could access summary information from 
all sites with which to validate ROC/P effectiveness statewide. 

The identified goal of the project was to design and pilot test a model 
MIS for ROC/Ps which would begin to meet local, association, and state level 
information needs. Committee members agreed that the ideal MIS would serve 
both as a management support tool for local managers and as a data base from 
which to report to the association and the state. The primary focus of the MIS 
should be to improve the effectiveness of local management. If the MIS 
did not meet immediate needs on the local level, it would be neglected or 
subverted and thus become ineffective in providing the state with accurate 
data. The primary goal of phase I was therefore to identify the 
information needs and prevailing conditions in local ROC/Ps and match 
these with the most appropriate system design features and development 
strategy for a model system. 

In narrowing the scope to a task which could be accomplished by a small 
research team in twelve months the MIS model was limited by committee 
action to data related to the life cycle of a course. The life cycle includes 
planning, implementation, operations, and evaluation phases. The unit of 
analysis selected was the course or the section, as opposed to the broader 
aggregate of the site or the smaller unit of the student. 
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The methodology chosen for the first phase of the study was a 
qualitative analysis, including group discussions, brainstorming sessions, 
unstructured interviews, and on-site observations. The sources of feedback 
information for the first phase of the study included the advisory committee, 
ROC/P leadership and representatives of special interest groups, individual 
managers and staff in selected sites, and existing information processing 
software currently in use. The following sections describe the both the major 
feedback soiurces and data gathering procedures as well as the insights and the 
impUcations for MIS design and development. 

Advisory Committee Input 

Following the initial meeting, the advisory conmiittee was convened 
three times during the first phase to review progress and set study directions. 
On September 20, committee members helped to plan and coordinate a 
brainstorming session for ROC/P directors attending the annual leadership 
Forum. In this planning meeting it became clear that the value of the session 
with the directors was to get local managers support of the system as a 
powerful force for the progress and development of an MIS. The MIS was to 
be designed with characteristics immediately appealing to locd managers. 

On November 7, the advisory conmiittee discussed the definition of data 
elements and explained the poUticized nature of the process for defining data 
elements for state reports. They suggested that the research team design a 
system in which data definitions could easily be modified. They also 
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recommended that the researchers avoid getting the system into the 
complexities of confronting state policies and ^i^stablished data definitions* It 
was clear &om these conversations that the MIS was not primarily about 
defining data elements* The implication of this insight was that the MIS must 
be desired in a way which would support local management decisions rather 
than impose data definitions* The wish of the conunittee was for a decision 
support system rather than a transaction processing system* 

On November 30, the advisory conunittee discussed the perception on 
the part of several ROC/P managers that the MIS was being developed as an 
evaluation tool for use by the state* The committee directed the research team 
to make all possible efforts to alleviate this misconception through the MIS 
documentation and design features and suggested that a one-page executive 
smnmary of the project be drafted (see appendix B)* The executive summary 
should emphasize the value of the MIS to directors as a decision*making tool* 
The insights from the advisory conmiittee are summarized in Table 3*2* 



Table 3*2 

Smnmary of Insights from the Advisory Committee 



INSIGHT 


IMPLICATION 


The MIS must meet local needs to 
be accepted by ROC/Ps* 


The MIS design must respond to 
locally felt management needs* 


The MIS is not primarily about 
defining data elements* 


The MIS should be more of a 
decision support tool for managers* 
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Input from ROC/P Directors and Management Groups 

During the course of Phase I, four different group sessions were 
scheduled for the piupose of acquiring input on the MIS data elements and 
features: 

1. ROC/P Directors at the Leadership Fonun in September 1990. 

2. Representatives in southern Califomia from various aspects of 
ROC/P management including finance, attendance, leg^ation, 
handbook, small schools, labor markets, public relations, 
curriculum, adult education and special needs populations in 
October 1990. 

3. Representatives in northern Califomia from similar management 
groups to #2 above in October 1990. 

4. ROC/P business officials at the Vocational Education Conference 
in November 1990. 

In each of these sessions the participants were given a brief overview of the 
MIS project and asked to contribute to the brainstorming and generation of a 
list of the data elements which managers would utihze in the defined phases 
of a course life cycle; planning, implementation, operations, and evaluation. 

The brainstorming sessions were especially fruitful in yielding lists of the 
data needs of ROC/P managers. In all three of the different types of groups 
broad themes and areas of data needs emerged rather quickly. Consensus was 
high on the need for data which could serve the following functions: (a) 
confirm labor markets, (b) validate curriculum, (c) verify links to business, 
industry, and higher education , (d) show cost efficiency, and document (e) 
instructional quality and (f) service to students (see document 2 in Su pporting 
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DnciiTnentfl for further detail)- The information needed by ROC/P leadership 
and by other management groups associated with ROC/Ps was clearly 
characterized as management decision information as opposed to simple 
reporting and control information. The insight arising from the group meetings 
is summarized in Table 3.3. 



Table 3.3 

Summaiy of Insights from Group Sessions 



INSIGHT 


IMPLICATION 


ROC/P leadership and others 
expressed needs for management 
information rather than mere 
report and control information. 


The MIS should function more for 
management decision support than 
for transaction processing and 
reporting. 



Input from Managers and Staff in Sites Visited 

Eleven individual ROC/P sites were visited during Phase I of the study 
in order to document current practices, clarify the issues which would help 
define the model MIS. The rationale for selecting sites to visit was based on 
the three factors of (a) exemplary management techniques in at least one of 
the four areas of planning , implementation, operation, or evaluation, (b) 
representation of the range of ROC/P types - coimty operated, joint powers, 
and single district, and (c) representation of the four regions of the state. 
Table 3.4 shows the distribution of ROC/Ps by governance type and by region 
with the sites visited. 
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Table 3.4 
Distribution of ROC/Ps by Region 
and by Governance Tyi)e with 
Sites Visited in Phrise I 



1 Type 


Region | 


Central 


Northern 


Coastal 


Southern | 


Totals 


County 


total 1 


6 


15 


10 


8 1 


39 


Operated 


visited 




2 




2 


4 


Joint 


total 1 


4 


2 


8 


11 


25 


Powers 


visited \ 






2 


3 


5 


Single 


total 1 


1 


0 


0 


5 


6 


District 


visited \ 


1 






I 


2 


Totals 


total 


11 


17 


18 


24 


- 

70 


visited \ 


1 


2 


2 


6 


11 



Prior to site visits ROC/Ps were asked to collect samples of a complete 
set of data-related documents. On site visits CERC researchers collected the 
requested data documents, toured ROC/P offices and facilities, and discussed 
data management priorities and practices with directors and other management 
team members. The analysis of data gathered through these visits was divided 
into three main subdivisions with a related question: 

1. The state of existing information systems - What do ROC/Ps have? 

2. Priorities for data collection - What do ROC/P managers want? 

3. Relationship of data to management - How do managers use data? 
Observations on each of these three questions are detailed in the sections that 
follow. 
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Observations on the What Systems ROC/Ps Have 

The major finding in regard to existing information systems is that there 
is no uniform system of transaction processing among the 70 ROC/Ps. A 
review of several existing transaction processing systems brought to light the 
fact that the time and costs associated with the development of a standardized 
collection and storage system for all ROC/Ps would be prohibitive in the near 
term. The high end ^stems in place in the largest ROC/Ps are far too 
expensive and complex to be attractive to the smaller ROC/Ps and a PC-based 
system for transaction processing is limited to the small and medium sized 
ROC/Ps. Clearly the type of system needed was a decision support system 
which could be designed independently fi:om existing transaction processing 
systems, but would be able to draw on information produced within these 
systems. A primary question arising from this determination was one of the 
most appropriate format. 

ROC/Ps vaiy widely in the extent to which automated information 
systems are in place, but all of the ROC/Ps visited have at least part of their 
operation computerized In all the visited sites office persoimel were equipped 
with personal desktop computers. Most sites had some connection to 
mainframe computers, usually in a county or district office. The lowest 
common denominator of automation for all the visited ROC/Ps was an MS-DOS 
IBM compatible desk-top personal computer (PC). The imphcation being, that 
a PC-based MIS could be made immediately available to virtually all ROC/P 
directors with minimal investment. 
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One of the areas explored to determine the state of existing information 
systems was an analysis of which types of information were in some 
computerized format. In all visited cases, some form of financial accomiting 
was electronically mediated In a majority of the sites accounting functions 
were performed on a mainframe computer. Attendance accounting systems 
were the second most prevalent automated systems found in the ROC/Ps 
visited. Two of these are reviewed in Supporting Document #8 (see preface) 
and found to be focussed primarily on data collection, storage, and retrieval, 
with less attention given to program quaUty evaluation/analysis or 
management decision support. Other information systems in the ROC/Ps 
included systems for course catalog production, for personnel management, and 
for equipment inventory control. In most cases where these systems were 
available, they were separate from one another both physically (in different 
computers) and organizationally (operated by different individuals). One of the 
greatest information needs, both observed and expressed by ROC/P managers 
was that of information integration. Managers v/anted to be able to look at 
information from a variety of sources on a conmion summary printout. The 
model MIS needed to integrate information from existing systems and 
elsewhere into a common analysis and reporting format which would support 
management decision making. This finding suggested that an "information 
analysis" type decision support system would be the most appropriate design. 
Table 3.5 sxmmiarizes the insights gained from observations about existing 
information systems in ROC/Ps. 
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Table 3.5 

Summary of Insights from Site Visits on 
The State of Existing Information Systems 



INSIGHT 


IMPLICATION 


No miiform transaction processing 
^^stem exists in ROC/Ps, nor is 
TPS development a feasible option. 


The MIS must be able io use 
information generated through 
existing TPSs. 


Most ROC/P managers have access 
to a personal computer. 


A PC-based MIS would be the most 
quick and economical to install. 


Most ROC/Ps have multiple, non- 
integrated information systems. 


An information analysis DSS would 
integrate diverse data systems. 



Observations on What ROC/P Managers Want 

Interviews with individual ROC/P managers confirmed the importance 
of many of the data ca ;egories which had been identified in the group sessions, 
however significant differences in priorities were evident from site to site. In 
one site the cost analysis data was of primary importance to the manager who 
had a rather elaborate reporting system which compared costs with ADA 
revenues generated on a monthly basis. In at least three other sites, managers 
said that measures of cost efficiency or cost effectiveness were not considered 
when evaluating the relative quahty of their courses. They felt that cost alone 
was not the best index for coiu*se comparison. 

In at least one site a great deal of emphasis was placed on the extent to 
which advisory committees were involved in coiurse analysis and improvement. 
Forms and other data gathering processes had been established to quantify the 
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activities of these committees. In another ROP placements were a chief 
indicator of course quality. A database of businesses and industries which had 
any relationship to the ROP was actively maintained Lists of graduate 
placements in local businesses were generated as feedback to instructors. 

In at least one large ROP a significant emphasis was placed on student 
opinion data collected in the follow-up. Former students were asked whether 
or not they had met their goals in the course and whether they would 
recommend the course to a friend In yet another site, an important measure 
of program success was a comparison of high school graduation rates among 
ROC/P students and non-ROC/P students. 

These differences in the priority of data collection needs expressed by 
different managers led to the three conclusions summarized in Table 3.6. 



Table 3.6 

Simunary of Insights firom Site Visits on 
Priorities for Data Collection 



INSIGHT 


IMPLICATION ^ 


Course quality measures are 
collected from a variety of sources. 


The MIS should integrate multiple 
course quality measures. 


ROC/P managers assign different 
priorities to different data 


The MIS should allow for variable 
weighting of a wide range of data. 


Little consensus exists on the 
relative importance of different 
quality indicators. 


The MIS should function as a 
'^representational*' system to model 
appropriate priorities. 
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Observations on How Data Are Used 

A primary goal of the MIS model was to improve the use of information 
by local ROC/P managers. To understand how the MIS could best achieve this 
goal it was necessary to docmnent prevailing conditions which prevent the 
optimimi use of information by managers. Four conditions were found to be 
impeding the effective use of course information by ROC/P managers: 

1. The course review process was approached negatively 

2. QuaUty documentation was limited to too few indicators 

3. Information was often available but not interpreted 

4. Quahty indicators were aggregated unevenly 

The fact that the comrse review process is largely a control mechanism 
to prevent misuse and abuse of state vocational education funds contributes 
negatively to the use of coiu'se quality information by local ROC/P managers. 
The Ed. Code S 52302.3 specifying how ROC/P courses are to be evaluated 
addresses three major criteria: (1) non-duplic:ation, (2) labor market demand, 
and (3) completions/placements. The avoidance of imnecessaiy duplication of 
training from one agency to another has apparently been a major concern of the 
state legislature, judging from the frequency with which it is addressed in the 
code and the level of detail with which it is described. The intent of the 
legislature on this point is clearly one of preventing inefficient use of taxpayer 
moneys through the establishment and operation of dupUcate training 
programs. 
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The second most prominent legislated criteria for course evaluation is 
that of a documented labor market demand. The governing bodies of 
ROC/Ps are enjoined by the code to work with local advisory conmiittees, 
Private Industry Coimcils, the Employment Development Department, and the 
California Occupational Information Coordinating Committee in documenting 
labor market demand The involvement of multiple agencies in developing 
protocols, the use of different coding systems for training programs and jobs, 
and the different levels of aggregation between labor market statistics and 
ROC/P courses I ive all contributed to a state of confusion as to how to apply 
this criteria in course evaluation. As with the non-dupUcation criteria, labor 
market documentation is intended as a control mechanism to prevent the use 
of taxpayer money for programs which do not prepare students for jobs. 

The code on course review also specifies that a course is to be of 
demonstrated effectiveness as measured by the emplojrment and completion 
success of its students. While these two indicators of course effects are 
probably among the more measurable and reportable, they are by no means 
suf&cient to document the value and quaUty of a course. They serve only as 
warnings to prevent the funding of programs which are not retaining students 
or which do not prepare students for job placement. 

The mandated course review process is characterized by negative control 
measures - indicators which warn that a problem exists and which mandate the 
consequence of course termination if the problem persists. Unfortunately the 
code provides Httle indication on how to positively select for the most effective 
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courses based on things like the quality of instruction, stability of the 
iy9nn^y^if;i n between the training prngram and industry, or positiye effects of 
training on the hves of students. Management by termination of low end 
exceptions may minimize inefficient and undesirable ROC/P programs, but it 
does not promote highly efficient and powerftd programs. It only promotes 
compliance-oriented data collection at the local leyeL 

The selection of ROC/P training programs on the basis of excellence can 
only be stimulated when indicators of excellence are used to compare courses 
with one another. While the control measures specified by the education code 
may be a necessaiy protection against problems, they are far from siifficient to 
promote excellence in vocational progranmiing. ROC/P managers must be 
given the tools by which a positive analysis of program quality can be locally 
controlled The MIS model should serve as just such an analysis tooL 

A second condition mitigating against the use of information by local 
managers is an inability to combine multiple variables in program quality 
assessment. Many ROC/P managers saw state mandated measixres of course 
quahty as overly simplistic and reductionist in nature. The quahty of an entire 
program cannot be determined by one or two indicators. Since no single 
measure can accurately portray the quahty or worth of a course, using only one 
or two measures in the high stakes decision of program termination is seen as 
grossly unfair. The more inferential the measure, the worse the inequity. This 
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problem of placing too high of stakes on a few inferential measures of quality 
was portrayed through three examples in discussions with ROC/P managers. 

1. When completions are the only measure of program quality, the 
tendency is to select for programs which are attractive and "fun" for students. 
Some training programs may have high attendance and completion rates, but 
may not be teaching students anything of substance. Reliance on completion 
rates alone could mask programs which amount to high*cost day care. 

2. When student placement in jobs becomes the sole measure of 
program quality, the incentive is to design programs which move students 
quickly and surely into placements. Some training programs may have high 
placements only because the jobs are those which students would have gotten 
without the training. A program in hamburger flipping illustrates an over- 
reliance on placement data* High placement rates may belie the fact that 
students may have just as easily gotten the job without training. 

3. If the cost per student is the sole measure of the quality of a 
vocational program, the incentive is to select for programs with the lowest 
possible cost and to fill all classes to their capacity. While cost efficiency is 
certainly a valuable goal in the administration of vocational training programs, 
training for more complex and higher paying jobs may well cost more than the 
training for the most simple and lowest paying jobs. Selection of training 
programs on cost alone will promote cheap programs which may be inadequate 
to addre&^ the need for workers in tomorrow's high-tech workplace. It also 
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disregards the responsibility of public vocational education to provide training 
for many levels of job complexity. 

Program quality is only achieved when the full range of quality measures 
are integrated into the selection process. The MIS should be a tool which wiU 
allow for more comprehensive and "fair" depictions and/or comparisojis of 
courses. It should allow for combined reporting of a large number of quality 
indicators which all contribute to a course section quality profile. 

A third major problem in utilizing data for management is one of data 
overload - having too much data and not enough time or energy to interpret 
it all. The decision making process, while it requires a rich variety of inputs, 
is not simply a case of generating a critical mass of evidence. The more data 
with which a manager is confronted, the more the need for pre-analysis and 
interpretation of the data. If the MIS is to function as a support for managers 
it should provide pre-analysis and interpretation of data. 

ROC/Ps are expected to be flexible and responsive to the changes in the 
labor market and student pool. This flexibiUty is achieved by being able to 
adjust programs quickly. A majority of the quick management interventions 
are targeted at the section level. A major problem arises when the data are 
not all aggregated at the section level. When cost information is 
aggregated only at the program level it is impossible to tell when a section is 
costing too much to run. If attendance information is aggregated only at the 
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course level it is not possible to tell if aU sections are pulling their equal weight 
in bringing in revenues. If personnel information is not section specific it is not 
possible to determine whether staff placement is at its most efficient. These 
and other major management decisions require that program quality indicators 
be collected and reported at the section level. The MIS model must highUght 
this need for section specific data collection and analysis . 



A siunmaiy of the insights and implications gathered through 
observations of the conditions precluding or preventing optimimi data use for 
management in ROC/Ps is presented in Table 3.7. 



Table 3.7 

Summary of Insights from Site Visits on 
The Relationship of Data to Management 



INSIGHT 


IMPLICATION 1 


Management by negative controls 
does not stimulate information use. 


The MIS should provide a positive 
course quality analysis. | 


Course quality documentation 
cannot be achieved with only one 
or two indicators. 


The MIS should allow for the | 
combination of multiple indicators 
in establishing a quality profile. 


The more data confronting a 
manager, the greater the need for 
pre*analysis and interpretation. 


The MIS should pre-analyze data 
and provide reporting which is easy 
for managers to interpret. 


Data aggregation at the section 
level affords the greatest support to 
managers in course management. 


The MIS should focus all measiures 
of course quahty at the section 
level. 
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Summary of Findings in Phase I 

Phase I of this project consisted of a literature review on the topic of 
MIS design and development as well as a field study to determine the 
conditions prevailing in ROC/Ps and the nature of the MIS needed. Two major 
decisions came out of Phase I of the study: 

1. the type of system to design 

2. the development strategy to pursue. 

The Type of MIS Design 

The type of MIS most needed and most feasible for developing in 
ROC/Ps was a decision support system (DSS). The specific type of DSS most 
appropriate to the conditions and needs in ROC/Ps was an analysis In formation 
system which integrates data fi-om disparate sv^urces into coherent analysis 
reports. Such a system facilitates both operational management and strategic 
planning related to ROC/P courses. It integrates and anal)rzes data on costs, 
enrollments, retention, attendance, ADA revenues, labor market, completions, 
placements, quality of instruction, student and employer feedback, and other 
measures of course quality. It helps local ROC/P management to (a) quickly 
identify those programs which most need their attention, (b) specifically target 
their interventions, and (c) pre ide documentation to both demonstrate the 
high quality of their programs and justify their management decisions. 



Page 44 CERC@UCR 10/23/91 

ERIC ^ 



The MIS also function as a representational system. The lack of clarity 
in the values and priorities of different indicators of course and program quality 
in ROC/Ps called for a system which would help to clarify the relative 
importance of various quality measures. A system of weighted details allows 
managers to model their decision making by rearranging weighting priorities 
mitil the sorted report reflects the ranking order they would have intuited 
The representational model empowers local managers to become users and 
controllers of information. It helps managers model the best mix of variables 
to include in the sensitive choices inherent in program selection and evaluation. 

The Best MIS Development Strategy 

Given the conditions and the needs clarified in Phase I, the selection of 
the most appropriate development strategy was clear. Selection was made firom 
among four options which were prominent in the Hterature - bottom-up, 
evolutionary, top-down, and middle-out. Because of the great diversity of data 
element definitions and electronic capabilities among ROC/Ps, a bottom-up 
approach is too expensive to develop and implement on a state-wide basis, and 
is not likely to increase the effectiveness with which ROC/P programs are 
managed. An evolutionary or modular approach does not sufficiently address 
the expressed need for integration of information firom a variety of sources for 
management use. A top-down approach is too time consuming to develop, and 
the changing nature of the data requirements in ROC/Ps would make the 
system obsolete before it could be completely designed. A middle-out or 
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proto-typing approach is the most appropriate strategy for system 
development because it: 

1. focusses on a particular management problem, 

2. can be quickly designed and implemented for pilot testing 

3. is flexible to changing data needs 

4. helps to further clarify the problem 

5. leads to the best long-term solution. 

A focus on the decisions regarding sections helps the system transcend the 
diversity of size, governance, and data use between the various ROC/Ps. 



Page 46 CERC @ UCR 10/23/91 

63 



Chapter Four 




Phase n: Design and Testing of the MIS Model 



Design of the MIS Model 

Phase II of the project was the developmental phase in which the 
findings from Phase I were synthesized into the first version of a model 
management information system. This chapter describes the various aspects 
of the design and testing process. In the first section the steps of the 
development process are outlined A description of the MIS model from the 
perspective of the user is provided in the second section. The third and fourth 
sections include details on the selection of pilot test sites and the procedures 
followed in the pilot testing process. The final section is a simmiary of the 
observations made during the pilot testing. 

MIS Model Design and Development Process 

The advisory committee and the research team, after careful analysis of 
the existing situation in ROC/Ps around the state and review of the possible 
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options outluied in the current literature on information systems elected to 
design an MIS which would function both as an ftTial ysia in fnrmati on decision 
support system and as a representational model As an analysis information 
DSS the system integrates information about courses firom several different 
sources to produce analyzed reports. As a representational model the system 
helps to clarify the values and priorities of local managers in the decisions they 
make about course retention, suspension, and termination. In reviewing the 
options for system development and implementation, it was decided that the 
middle-out, or proto-typing approach would best fit the situation. 

The proto-typing approach to system development begins with the quick 
creation of a small, workable, modular system which is focussed on a particular 
problem or set of problems. It is implemented next to existing systems. The 
users provide feedback to the designers on the fit of the system features to 
their situation. Designers incorporate changes into subsequent versions of the 
model which are further tested and critiqued. This reiterative cycle continues 
with refinements and expansions of the system as long as the development is 
beneficial to the users and/or cost effective for the supporting agency. 

The strategy used to design the proto-type model was that of decision 
analysis. Decision analysis was particularly suited to the situation with 
ROC/Ps because decisions about courses were much more uniform across 
ROC/Ps than were data elements or data collection protocols. A focus on 
decisions was the most logical way to achieve a conunon system among such a 
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diverse group of organizations as the ROC/Ps. The decision analysis strategy 
is comprised of five major steps leading up to pilot testing (Tran, 1986): 

1. Define the decision areas on which to focus 

2. Identify key decisions which contribute to effective management 

3. Classify the decisions and the information which supports them 

4. Develop a decision model 

5. Suggest data to be used in the pilot test 

Each of these steps to the decision analysis strategy for prototype design is 
briefly described in the following paragraphs. Supporting materials can be 
found in the Supporting Documents publication (see preface). 

1. Define Decision Areas. Early in the study the advisory committee had 
narrowed the focus to the management information needs of local 
managers and Umited the scope to the life cycle of a course. A course was 
defined as an approved curricular unit characterized by listed objectives and 
competencies to be achieved within a given instructional framework in a 
specified amount of time. The life cycle of a course was defined from the 
perspective of ROC/P managers as the four stages of planning, implementation, 
operation, and evaluation. The decision areas were therefore confined to those 
decisions made regarding a course from its beginning to its termination. 

2. Identify Key Decisions. A flowchart of key decisions was developed 
ia which the data needs and sources for each decision were plotted This flow 
chart was printed in Supporting Documents #5 (see preface). The decisions in 
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this chart were phrased in terms of a question to be answered For example^ 
*ls the labor market demand sufficiently strong that we can expect placement 
of graduates into the jobs for which they are being trained?" An analysis of the 
decisions outlined in the flowchart revealed that the msgority were concerned 
with documenting course quality. 

3. Classifir Decisions. The information needed to support management 
decisions regarding course quaUty were classified into four major domains or 
possible sources of feedback. 

a. ROC/P managers and instructors, 

b. students and graduates of ROC/P courses, 

c. high schools and colleges 

A employers of ROC/P graduates. 
Within each cell made from the overlapping of the four domain circles on one 
another are subcategories of information about course quality. For example, 
at the intersection of students and ROC/P management is found the category 
of instructional quahty, showing that feedback on instructional quality ^s jointly 
determined by the students and the ROC/P administration. These domains of 
coiu^e quahty measurement and analysis are illustrated in Supporting 
Documents #6 (see preface)* More specific examples of the information needs 
related to these domains are Usted in Su pporting Documents #7 (see preface). 

4. Develop Decision Model. A computer model for decision support was 
developed in response to the guidelines developed in phase I of the study and 
based on a model for problem solving forwarded by Carkhuff and Anthony 
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(1981). At the core of CarkhuSf and Anthony's model is a sorting function. 
Course sections are sorted in rank order &om best to worst depending on 
a positive, neutral, or negative rating on multiple weighted factors. The system 
is flexible in a number of ways. An unlimited number of details may be defined 
to measure different aspects of section quality. Detail standards which define 
the ranges for the positive, neutral, or negative ratings of values can be 
modified interactively on screen. Weights can be assigned to the different 
details included in a given report. These weights determine the impact of each 
detail on the section's relative rank score. These weights can also be eauly 
manipiilated imtH they approximate the intuitive relative values that managers 
assign to different measures of course quaUty. A technical description of the 
computer model is provided in SuPDorting Documents #9 (see preface). 

5. Suggest Data to Use. The advisory committee met in February, 
1991, to discuss possible detail data to require in the pilot test of the model 
From the diagram of domains of course quality, three indicators were selected: 

1. 'Direct Cost per ADA" defined as all the direct costs for a section 
divided by the total ADA the section generated. 

2. 'Percent of Attendees Certified" defined as the total number of 
students certified di\dded by the total number attending. 

3. 'Percent of Positive Exits" and was defined as the total number of 
students who either continued in school, got a job, or joined the miUtary 
divided by the total number attending. 

Suggested parameters for these details are found in Appendix A - a worksheet 

for detail definition. 
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The MIS Model Software - A Decision Support System (DSS) 

The software for the ROCP-DSS was designed to be user-friendly- It is 
largely menu«driven. Users select from a menu to move around within the 
modules of the program. The Main Menu screen showing the File Maintenance 
menu is illustrated in. Figure 4.1. 




Figure 4.1 
DSS Main Menu Screen 



The first five steps in the use of the system are Usted as optional maintenance 
modules in the file maintenance menu. Fidl program use involves six steps: 
L Create a database of coiurse numbers and titles 

2. Create a related database of course sections 

3. Define details or measures of section quality 

4. Enter values for each detail on all sections 

5. Define analysis report formats 

6. Run analysis reports and repeat step 5 as needed 
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The ROCP-DSS Manual (Dick, 1991) describes the various menus and 
procedures in greater detail. 

The major function for which the DSS exists, running an analysis report, 
is only possible when the first five steps have been successfiilly completed. 
Once the databases for courses, sections, details, values, and analysis reports 
are in place the real power of the system becomes evident. Modifications can 
be made to the reports immediately prior to printing. This feature allows for 
asking and answering "What iT questions by changing detail weights and other 
parameters. This feature is particularly important in giving the DSS the 
distinction of functioning as a representational model. If managers '^lay 
around" with the mix of the details and weights for their reports until the 
sections sort in the "right" order, then the report actually begins to represent 
the internal management intuitions of ROC/P managers. The imphcations of 
the use of such a representational model as a research tool for the study of 
management decision making process can not be understated The DSS is 
designed primarily as a management assistance tool however. A brief review 
of what the report looks like and how it can be used will illustrate the value of 
the DSS for this purpose. 

Interpreting and Using the Analysis Report 

The decision support analysis report is printed in two sections. Samples 
of both parts of the analysis report are shown in Figure 4«2. The first is a 
sorted listing of all the sections included in the analysis. The list includes a 
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matrix of sections by details showing the relationship of each section to the 
standards which have been set on each detail. For example a plus sign shows 
that the value for a detail is within a defined positive range. 
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Figure 4.2 
Sample DSS Analysis Report 
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The second part of the report is shown as page 2 in Figure 4.2. It is called the 
Report Summary and includes descriptive information on the detail parameters 
such as relative weights and the low and high accept values which define range 
limits for positive, neutral, and negative. The sunmiary also includes statistics 
about the values on each detail such as the Tninimum, mean, and maximum 
values and the approximate values at the 25th and 75th percentiles. 

The analysis reports can be used in a number of different ways in the 
support of management decisions. Several suggested uses of the reports 
include the following: 

1 . Assisting managers in making decisions to retain, suspend, or drop 
courses/sections. 

2. Justifying course/section decisions to superintendents or boards. 

3. Providing feedback to mid-level managers to help focus 
interventions and instructor inservices. 

4. Providing feedback to instructors on the relative quality of their 
class sections. 

5. Providing report data to be shared among the other ROPs in the 
association. 

6. Developing a database from which state reports are generated 

7. Supplementing or replacing the course quality review process. 
The audience for analysis report information is by no means restricted to the 
ROC/P manager, as can be seen from the preceding list. Individuals at all 
levels in the organization can benefit from a comparative quaUty analysis of 
course sections. The ROCP-DSS Instruction Manual provides a much more 
detailed description of system use and benefits (Dick, 1991). 
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Pilot Testing of the MIS Model 



When the fibrst version of the MIS model software was available in late 
March, 1991, it was shipped, along with the manual to twelve ofQcially selected 
pilot test sites. The pilot sites were given from three to four months in which 
to evaluate the software for its usefulness and appHcabihty to their situation. 
During this time the sites were monitored periodically for their progress in 
implementing the pilot test. In the next sections three major topics are 
covered regarding the pilot testing: the selection of pilot sites, the 
procedures followed during the pilot test, and observations made regarding 
implementation of the software by the pilot sites during the testing phase. 

Selection of the Pilot Test Sites 

The selection of ROC/Ps to participate as pilot test sites was based on 
four criterion: 

1. a willingness to participate as a pilot site, 

2. governance type representation, 

3. geographical region representation, 

4. size representation. 

Each of the selection criteria is described in greater detail in the following 
paragraphs. 
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Willingness to participate. 

A willingness to participate as a pilot test site was an important criterion 
in the selection of sites for two reasons. Given the limited time and resources 
available for the study, the research team did not want to take on the extra 
burden of working with xmwilling players. In addition, a process of self- 
selection was expected to more accurately reflect the voluntary adoption 
patterns expected in later implementation. The managers of ROC/Ps were 
surveyed for their willingness to be involved in the MIS study. Twenty ROC/P 
managers expressed a willingness to serve as pilot test sites. These willing 
sites adequately represented the other three criteria of governance type, 
regional distribution, and relative size. 

Representation of Governance Types . 

Three different governance types of ROC/Ps exist: 

1. single district 

2. joint powers agreement (JPA) 

3. coimty office operated 

At present the coimty operated ROC/Ps are the most prevalent, with thirty- 
nine in operation around the state. Two of these have a specifically designated 
training center (ROC). Of the twenty-five joint powers ROC/Ps, only five are 
ROCs or ROC/Ps. Only six single district ROC/Ps exist in the state, three 
large urban districts in the Los Angeles area, and three geographically large 
and isolated districts. Two single district operations have a training center. 
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The steering committee selected six county operated, four joint powers, 
and two single district ROC/Ps as part of the pilot study to insure a 
representation of all three types. In addition the team stipulated that at least 
one of each of the three t3^es sampled would be an operation with a designated 
training Center. Figure 4.3 shows a comparative count of the ROC/Ps by 
governance type along with a relative coimt of the sample by governance type. 
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Figure 4.3 

ROC/Ps and Pilot Sites by Governance Type 
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Representation of the Foiir Regions. ROC/Ps are unevenly distributed 
in the four state regions. The central region has the fewest ROC/Ps, only 
eleven, but with the ftdl range of governance types. The northern region has 
seventeen ROPs, all but two of which are county operated The coastal region 
has eighteen ROC/Ps, ten of which are county operated The remaining eight 
are JPA. The southern region has the greatest number of operations with 
twenty-four ROC/Ps. The majority in the south are JPA ROPs. Most of the 
single district ROC/Ps are a part of the southern region. Figure 4.4 shows the 
distribution of ROC/Ps by region and by type. 



Distribution of ROC/Ps 
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Figure 4.4 
ROC/Ps by Region and Type 
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At least two sites per region were selected to participate in the pilot test. Four 
additional sites were selected &om the southern region because of its greater 
niunbers and because of the proximity of these sites to the research 
headquarters in Riverside* 

Representation of Size. The nine largest ROC/Ps represent only about 
thirteen percent of all ROC/Ps in the state, but account for over fifty percent 
of the ADA generated by all ROC/Ps. In selecting a sample to represent the 
different sizes of ROC/Ps, it was important to include large and small as well 
as several in the mid range on size. Four of the of the top nine lare:est 
ROC/Ps and a representative sample of the mid to small sized ROC/Ps W v;re 
included in the sample. Several urban ROPs which represent a rather small 
geographical area and several rural ROPs which span large areas were also 
involved At least one ROC which operates like a technical high school was also 
included in the study. The least well represented size of ROC/Ps was the 
smallest (fewer than 600 ADA). This was not an oversight on the part of the 
committee, but was done with the thought that the system could be more 
easily scaled down to fit the smaller ROC/Ps than it could be scaled up to fit 
larger ones. The primary target group was therefore the mid to large sized 
ROC/Ps. Figure 4.5 shows the distribution of ROC/Ps by size of ADA along 
with the distribution of the pilot sample. 



Page 60 



CERC@UCR 10/23/91 



Siieof 



ROQP 



ROC/Ps by Size of ADA 

ai^ pilot sitses mm each size tange 




22i 



5 2 * u IS it S 2*7 



Count 



30 



Figure 4.5 
ROC/Ps and Pilot Sites by Size 



Pilot Testing Procedures 

The twelve pilot sites were originally divided into two groups, those 
which were likely to need technical assistance from the researchers and those 
able to implement the MIS model using in-house resoiu'ces. This distinction 
was made primarily to alleviate the time and travel pressure from the research 
team and to encourage the pilot sites to work independently. All of the twelve 
sites were otherwise treated in basically the same maimer throughout the 
testing process. The procediu-es included notification of selection as pilot site, 
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distribution of software and documentation, calls to verify receipt of software 
and to notify of the availability of assistance, site visits to assist with technical 
and/or conceptual dif&culties as needed, and scheduling of final debriefing 
interviews. Each of these procedures is explained in greater detail in the 
following paragraphs. 

Pilot Sites Notified. After the February meeting of the advisory 
committee, in which the twelve pilot sites had been recommended, the selected 
sites were notified The MIS project manager, a former ROC/P manager, 
contacted each of the twelve pilot site managers personally to welcome them 
into the study as a pilot test site. Once the formal introductions had been 
made, the responsibility for carrying out the pilot testing was left to the 
member of the research team in charge of field contacts. 

Software and Docunn entation Distributed. On March 19 the available 
software package was loaded onto IBM compatible personal computers in two 
of the test sites. On March 28 it was loaded onto a computer in a third site. 
In all three of these first sites the software loading was done or supervised by 
the field researcher. Convinced that the loading operation was simple enough 
and well documented in the manual and that the program was working as 
intended, the research team mailed the software and manual to the remaining 
nine sites on April 3. 
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An updated version of the software (v. 1.3), including the completed 
analysis reporting module, was distributed on disks by May 15* In the last 
week of May the Manual chapter covering the analysis report module was 
distributed. As feedback on the software program began coming in &om the 
pilot test sites several modifications were made to the software program. 
Several cumbersome aspects of the program were streamlined, a few new 
featiures were added to increase both program efficiency and reporting power. 
The disk with this update (v. 1.4) was mailed on Friday, June 14. Version 1.4 
is the latest reiteration of the program at the time of this report's printing. 

Phone Contacts Established. Within a week of first mailing the MIS 
model software and manual the field researcher contacted each site to confirm 
receipt of the mailing. Researchers estabUshed a schedule for contacting each 
pilot site at regular two-week intervals to monitor progress. All sites were 
notified in the first week of May about a workshop on use of the MIS model to 
be conducted at the May 9th Vocational Education convention in Oakland At 
the end of May and into the first week of Jxme aU pilot sites were contacted to 
confirm the receipt of the analysis module of the software and the 
corresponding chapter inserts for the manual. 
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Trouble-shooting Site Visits Conducted The researcher made site visits 
to eight of the twelve pilot test sites during the course of the pilot test period 
In some cases these visits involved basic technical assistance in data entry and 
report generation. In other cases the discussion was much more on a 
conceptual level with the researcher explaining the utility of the program to 
the managers at a site. All eight site visits took place between May 20 and 
June 14, In each of three county sites, the researcher worked either alone or 
with a secretary/registrar to make data transformations and/or enter the data 
in to the MIS model database. Once the database was filled with values, the 
researcher assisted the secretary in defining several different analysis reports, 

Foiu: site visits consisting mostly of conceptual discussion and 
demonstration of the software. All three of these sites were large urban ROPs, 
two county operated, and one JPA, In two of these sites the researcher spent 
time inservicing secretaries and/or data persons on the program. In neither 
case were managers participating in the inservice. In another large ROP the 
researcher met with a team of higher level managers to demonstrate the 
program and discuss issues regarding its use. The fourth site which received 
conceptual help had two data processing personnel who had already 
experimented substantially with the software and wanted to clarify some 
conceptual and technical difficulties during the site visit. 
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Summary of Observations from the Pilot Test 

Observations during the pilot testing phase of the study led to inferences 
regarding both the technical and conceptual implementation of the DSS 
software. The following Ust is a sununary of these observations. Each item is 
explained in further detail in subsequent paragraphs and revisited in the 
summary chapter. 

Technical implementation 

1. The software was relatively free of *hugs" 

2. Technical documentation was clear and easy to follow 

3. Current data aggregations were problematic 

4. Data preparation and input time was slow and cumbersome 

Conceptual implementation 

5. Top-manager involvement was lower than optimimi 

6. Conceptual imderstanding was difficult - more training needed 

Initial observations on the technical implementation of the MIS model 
software were that it was surprisingly fool-proof and free of software bugs. 
Other than the few problems related in an earher section, most of which were 
addressed and corrected in update version 1.4 of the software, the program 
seemed to run just as intended and the documentation was clear enough to 
be followed by any but the most computer phobic among ROC/P managers and 
office staff. 
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A common problem encoimtered in DSS implementation was that the 
actual numeric values associated with the suggested details were not readily 
available in the format or the aggregation called for by the software. Both 
the completion and placement data were frequently aggregated only at the 
program or course level rather than at the section level Costs per ADA v/ere 
in some cases not already precalculated, so long-hand division was needed to 
transform the raw data into the correct format for the values fields. To the 
. extent that ROC/Ps collect data only to satisfy state requirements for broad 
sunmiary data, they miss the usefulness of the data for fine tuned local 
management decision support. Collecting summary data exclusively for 
reporting carries with it Uttle motivation to keep the data accurate and reliable. 
The DSS software brought this problem into clear focus in several sites, 
ROC/Ps need to be made aware of the need to collect as much data as possible 
at the section level early in the implementation process. Future versions of the 
software needs to incorporate routines for aggregating and desegregating 
certain types of data automatically. 

The data collection, transformation, and entry processes necessary for 
DSS use were found to be particularly time consuming and tedious. The 
definition of analysis reports was rather difficult to conceptualize without 
having seen a completed report. Certainly the limitations of the current test 
version of the program are obvious, not the least of which is the fact that all 
the data must be collected and transformed outside the program itself, A m£gor 
point of focus for a production model of the software would be a much more 
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sophisticated electronic interface with existing data systems which would 
automatically import existing data and transform it appropriately. 

The implementation of the software proceeded most quickly in sites 
where the managers in charge were computer Uterate enough to be able to 
conceptually grasp the utility of the MIS model software, understand the 
necessary tasks involved from a quick perusal of the documentation, and had 
computer Uterate staff with whom to collaborate. A primary component of 
delayed or non-implementation seems to have been a manager who delegated 
responsibility without a personal interest and follow-through. In general the 
participation of managers was lower than expected 

Part of the problem of non-involvement of managers was the difficulty 
in conceptually understanding the major function and uses of the modeL 
Reorganizing the documentation based on a more carefid task analysis may be 
one way to increase future tmderstanding and implementation. The software 
manual should provide separate instructions for technical stafT who will be 
loading the software and configuring it for use in the office, clerical staff who 
will be entering and updating the course and value data, and for managers who 
must define details and analysis reports. Also, a clear presentation of the 
outcomes and benefits of using the software should be incorporated into 
training materials and sessions mounted to both precede and to complement 
the implementation process. 
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- Phase ni: Data Collection, Analysis, and Findings * 



Phase III of the project involved three basic processes, data coUection, 
analysis of the data, and presentation of the Hndings. The data collection 
process was through a structured interview. The analysis process involved 
sorting interview responses and observations into categories, rating various 
aspects of the implementation, and deciding on the relative importance of 
various pieces of data. The findings were divided into three categories, the 
implementation process and it's success, evaluations of the MIS model 
characteristics including criticisms, suggestions for improvements, and 
commendations, and the perceived uses of the system . 

The original pilot site sample included twelve ROC/Ps, Three additional 
ROC/Ps requested and received copies of the DSS software at the time of 
distribution. One of these additional sites, a joint powers ROC/P, participated 
with sufficient interest to be included in the final debriefing interviews and 
analysis. The total n for the data presented in this chapter was thirteen. 
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Data Collection: Debriefing Interview 

Pilot sites were asked to generate three different analysis reports using 
the DSS software, fill out course and section difference forms for each of the 
courses and sections sampled, and download electronic copies of the data fQes 
created Each site was also expected to participate in a 2-3 hour structured 
debriefing interview with the researchers. 

A six page foim was developed for debriefing interviews (^pendix B). 
The interview instnmient was designed to capture three different kinds of 
information: descriptive information about the pilot sites, narrative 
information about the process and extent of MIS model implementation, and 
appraisal information related to the problems and possibilities of the MIS 
model software. Descriptive information about the pilot sites included data on 
administrative structure, relative size, service area characteristics, technical 
capabihties, personal and office characteristics relative to information use, and 
the ways in which different data elements (details) were defined. Narrative 
information about the implementation was collected from each individual who 
participated in any aspect of the pilot test. Narrative data was categorized 
under the headings of personnel involvements in the pilot test, analysis reports 
generated, and interpretation of reports, ^praisal information was collected 
throughout the intenriew with questions about the inhibiting and enabling 
factors to implementation, but was specifically addressed in a section on the 
future uses of the DSS software. 
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Analysis: Extent of Implementation 

Implementation of the software was conducted on a completely voluntary 
basis by the pilot sites. It was expected that the levels of implementation 
would differ from site to site. To establish the level of implementation 
achieved by the different sites, three independent measures were developed for 
analyzing the debriefing interview data. These measures assessed (1) the 
extent to which the prescribed tasked had been completed, (2) the level of 
involvement of the ROC/P director or an upper level manager in key 
processes, and (3) the level of conceptual understanding of the system by the 
director or manager at the debriefing interview. 

1. Completion of Prescribed Tasks , The first measure of 
implementation assessed the extent to which the prescribed tasks of software 
testing had been successfully completed The sites were rated on a five point 
scale on the three major technical aspects of conducting the pilot test: 
(a) creating a sample database of courses and sections, (b) defining details and 
analysis reports, and (c) producing analysis reports. The highest available 
score on task completion was twenty. One pilot site received a score of twenty. 
Six sites received scores ranging from twelve to eighteen. Three sites had 
accomplished less than the TniTiiTniini tasks described in the software manual 
for pilot testing. The remaining two sites which had not entered course and 
section data into tha^ system received scores of zero. 
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2. Involvement of Manager . Besides the technical inspection of the 
software, an important objective of the pilot testing process was the analysis of 
its usefulness as a management tool To fairly test the system's usefulness as 
a tool for ROC/P managers it was necessary that managers to be fully engaged 
in the pilot test process. The software manual specified that managers should 
be involved in three aspects of the pilot testing process: (a) the definition of 
details, (b) the definition of analysis reports, and (c) the reviewing of printed 
analysis reports with and eye to needed modifications. A simple scoring system 
was established to rate the extent of involvement of ROC/P directors and/or 
other top level managers. The maximimi available score on the measiure of 
involvement of an executive manager was ten. Three sites received scores of 
ten. vOne site received a score of eight. All of the other sites received scores 
of five or less. In half of the sites the top level manager had not been involved 
in any of the three key aspects of model use^ 
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3. Conceptual Understanding of the System. Since a major objective of 
the pilot testing of the DSS software was to determine its value to ROC/P 
managers as a management tool, it was necessary that at least some of the 
managers of the Pilot test sites have a fairly clear conceptual grasp of the 
nature of the system. Without a conceptual understanding of the system it 
would be impossible to make fair judgements about its value to them. The 
answers to three of the questions in the debriefing interview form were 
anal3rzed and scored to establish a numerical indication of the extent of 
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conceptual understanding of the system by managers and directors. The three 
questions assessed the abiUty of the managers to (a) generate additional details 
appropriate to the system, (b) suggest possible uses and applications of the 
system and its reports, (c) describe changes to the system which would 
enhance its usefulness as a management tooL The scoring was on a range of 
zero to five for each of the three categories for a maximum of 15 points. 

Although none of the sites received the top score of fifteen points, eight 
of them had scores of ten or more. High conceptual understanding scores even 
in sites with low task completion and low manager involvement indicated that 
these were not necessarily prerequisite to a tolerable conceptual understanding 
of the system by a manager. 

Findings I: Factors Related to Implementation 

In the following paragraphs several factors, including the governance and 
administrative structure, size, technical capabihties, level of assistance &om the 
researcher, and the administrative level of the main person involved in the 
pilot testing are examined for their relationships to the different measures of 
pilot test implementation: (1) task completion, (2) manager involvement, and 
(3) conceptual understanding. 



Factors Related to Task Completion . None of the factors of governance, 
location, or level of person conducting the pilot test were apparently related to 
the level of task completion. The factors of size, commitment of director to the 
pilot, computer configxurations, and staff size all had some bearing on task 
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completion. PUot sites with more staff and computer resources generally had 
higher task completion levels. The top five sites in terms of task completion 
were medium sized (between 500 and 2000 ADA). Sites with high task 
completion seemed to take the pilot test more seriously. The top five sites 
each entered more than 40 courses each into the system for pilot testing. The 
remaining sites had each entered samples of fewer than 30 courses for testing. 

Factors Related to Director or Manager Involvement . Governance, 
numbers of office staff, and the computer Uteracy of the director all seemed to 
play a role in the extent of manager involvement in the pilot test. Managers 
in the joint powers ROC/Ps were much more likely to be involved in defining 
details and analysis reports than were managers in the county operated 
ROC/Ps. This difference was explained with an analysis of the difference in 
the numbers of office staff. JPA offices averaged significantly fewer office staff 
than county operated offices. Size may have also been a significant factor. In 
the three largest sites, with more than nine administrative staff and more than 
thirty classified office staff, manager involvement was the lowest. In the 
smallest offices, with one to three administrators and fewer than five classified 
staff managers were only sUghtly more involved. The four sites characterized 
by the greatest manager involvement all had three to six administrative staff 
and five to eleven classified staff. The computer literacy of the manager was 
another important factor in manager involvement. All of the more involved 
managers were computer Uterate and used a PC regularly in their work or at 
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home. Of the three sites with the least involvement of a manager, two of the 
site directors were admittedly computer illiterate. 

Factors Related to a Manager's Conceptual Understanding of the Model 
The three factors clearly related to manager's level of conceptual understanding 
were (a) the level of their involvement with implementation, G)) their level 
of their exposiu*e to training, and (b) their relation to the chief executive of the 
ROC/P. In all cases where a manager had been highly involved in the pilot 
test conceptual understanding was high. Others with high conceptual scores 
had spent extensive time exposed to the training of the researchers. This 
finding suggests that managers become familiar with the system either through 
involvement with the s^oftware or through exposure to training. 

The other factor closely related to understanding was the position of the 
person most responsible for conducting the pilot test. In the top ranked site 
the top manager took direct responsibiUty for the pilot test. In all twelve of 
the other sites, the task of pilot testing had been delegated to someone under 
the ROC/P director. In the high ranking cases the manager responsible for the 
pilot test worked closely with the executive director as part of a top 
management team. In the lowest ranking cases the responsibiUty for pilot 
testing had been delegated to secretarial, data processing, or clerical staff. 
Since the DSS model was not designed as a transaction processing tool to 
improve the clerical efficiency of an ROC/P office, those who treated it as such 
failed to understand the power of the DSS as a management tool. Clearly a 
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m£gor hurdle to a broad-based understanding and acceptance of the DSS model 
for ROC/Ps is an education of managers to the fundamental differences 
between transaction processing systems and management information systems. 

Findings 11: Evaluation of the DSS Model Characteristics 

The evaluation feedback from the pilot sites regarding the DSS model 
characteristics was divided into three categories: 

1. criticisms 

2. suggestions for improvements 

3. positive features of the system 

Criticisms of the DSS Model 

When asked if they would use the system in its current state only two 
of the pilot sites responded with an outright no. Both cases were large 
operations (over 4000 ADA). One was a site in which a large, comprehensive 
information system had only recently been put into place and was already 
producing reports similar to those generated by the DSS. The other was a site 
in which the management was in search of an information system for 
transaction processing and record keeping. In both cases, the persons who 
reviewed the DSS model had expectations that it was going to function as a 
transaction processing system for data storage and retrieval. This expectation 
was evident from the following comment: 
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■ The DSS provides inadequate support for year-to year planning^ 
and no support for on-going operations. The system does not 
contain the information necessary to generate appropriate 
management information reports* 

The beUef that information systems should be developed from the bottom up 

was most evident in these two sites which said they would not use the DSS in 

its current form* The system evaluators disagreed with the basic premise in 

the DSS approach that it is possible to separate data collection and storage 

from data analysis and reporting as evidenced in statements like: 

■ There are Uterally thousands of details that must be recorded, 
tracked, reported, and analyzed just to operate an ROP. An 
effective MIS must first manage these details. An effective MIS 
must become the repository for the ROP's operational 
information, with the paper files used as supporting documents. 



Another criticism of the DSS was that the reports coxdd expose 
individuals or the ROC/P to imfair comparisons or unwanted scrutiny. Several 
sites expressed concerns along these lines. 

■ My most serious reservation is that the state would lock into one 
or more details for unfair comparisons of ROPs. 

■ Fm afraid of very negative and defensive reactions to the DSS. 
The reports may create alienation among the different managers 
in the office. 

■ The reaction of several members of our management team to the 
analysis report was one of caution. They were afraid of letting 
the reports get into the wrong hands. Specifically they did not 
want board members to see the reports. 

■ You are likely to meet the greatest resistance to the DSS among 
the business and attendance accounting personnel (if they feel the 
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importance of their data is being diluted by other data, or if they 
find that they are losing control over the interpretation of data). 

These comments indicate a need for a better system of program quality analysis 

in ROC/Ps. An important ingredient in the establishment of such a system 

must bo localized control. Unless and until local manager can discover the 

strengths and weaknesses of their programs in a setting where they have both 

the opportunity and the power to make corrections and improvements, they 

will be continue to be resistant to the use of information which could be used 

in connection with sanctions or negative comparisons. A major step in the 

promotion and development of the DSS for ROC/Ps is to give local managers 

a fair amount of time to experiment with the system without having to make 

their findings public. Without this time and privacy cushion it is unlikely that 

the ^stem will ever get beyond an automated report generator. 



By far the most common criticism of the model was labor intensive 
nature of the initial data input. This criticism was partly related to the 
expectation that the system was going to function as a transaction processing 
system in making data processing more efficient. It was also legitimate 
criticism of a calculated weakness of the first prototype. With the give time 
and resources for developing a model, efficiency of data collection was sacrificed 
in favor of eloquence of data analysis. At least one site made significant 
progress in overcoming the problem of slow data entry by doing a direct 
electronic transfer of course and section data. In further developments of the 
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model the data input eflficiency would be overcome through electronic linkages 
to existing transaction processing systems* 

Su ggestions for Improving the DSS 

Several sites noted the problem of strict field definition as hampering the 
direct transfer of course and section data and suggested that using less 
restrictive fields for program, course, section, and location codes would 
solve this problem: 

■ Allow ROPs to enter their own program, course, section, and 
location nodes. Make the fields for these less restrictive in size 
and type 

■ Course number fields should allow alpha characters, not just 
nmneric. 

Another site suggested changes in the order of data entry as a possible way to 
simplify the process* 

■ It could save time and energy if the sections were defined at the 
same time as the courses. 

Clearly, among the first additions to subsequent versions of the DSS software 

must be modules which will make the electronic transfer of data into the 

system a matter of selecting from a menu* The overhead costs of data entry 

must be cut significantly if the DSS is to be a tool which will be regularly used 

by ROC/P managers. 

Several su^estions were given on ways to improve the user interface of 

the DSS software. Technical fixes suggested were to allow for changes of 
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the screen color combinations so the program could be modified to work on 
monochrome monitors, and to use function keys to shortcut some of the 
processes. Other improvements of the user interface included suggestions to 
speed up the process for search and update of data records, to provide 
more automatic checks of data accuracy, and to allow for viewing and 
reporting the data in different ways* 

■ SimpUfy the procedure for changing or updating values data and 
provide feedback that data has been changed 

■ Would Uke a quick way to access individual records using find 
commands or english term queries. 

■ Need a screen which prompts whether the direction is accurate - 
to be sure the positive and negative signs are appropriately 
labeled. 

■ A production system should be mouse driven and more 
interactive. It should have validation screens for checking the 
accuracy of raw data. 

■ We would like more optional screens and ways of viewing the data 
such as a browse mode or an on-screen review of an individual 
section profile with the raw data for all details. 

■ Vary the look of the printouts. They all look too much the same. 



Other suggestions for system improvements had more to do with the 

function of the system. Several reviewers pointed to a need for the system to 

be able to calculate new details from existing details, or to aggregate or 

disaggregate data which applied to courses or sections. 

■ Provide the capabiUty to automatically calculate a new detail fi*om 
two or more existing details. 
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Provide a way to deal with changes that deal with more than one 
section such as when a teacher salary goes up and impacts the 
cost of several sections. 



An even more complex process requested by those testing the system 
was that of trend analysis and reporting. 

■ Provide for tracking of change over time - trend analysis. 

■ Consider a 3-5 year trend analysis with exception reporting - for 
example to show those with 4 out of 5 years below standard 

Some of the reviewers felt that the ^stem should provide more on-line 
help and more thorough explanations of the benefits of the system. 

■ The manual for a final system will need substantially more 
information on how to define details and why, and how to use 
reports. 

r 

One or two of the sites even proposed that the system could eventually become 
an expert system which could suggest solutions to problems it found 

■ Eventually the system could have help screens which would 
suggest possible interventions to problems which have been 
identified 

In using the system as a coiurse evaluation tool one site poir.ted out the 
importance of using positive terminology, while another site suggested that the 
system should be more interactive with opportunity to insert memos and 
include these in reports as a way to conduct a course by course improvement. 
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■ We would avoid the use of the terms evaluation and termination. 
They have negative connotations. We woxild suggest terms such 
as "quality building*" and '"partnerships for program improvement.'* 

■ It would be nice to be able to insert comments (justifications) next 
to certain low detail values on some courses or sections. These 
could be combined with a summary of actions to be taken or 
suggested interventions in a p'^inted report which would profile an 
individual section or course. 

Other suggestions on how to improve the system included the addition 
of more section and course code fields on which to select for analysis, the 
addition of more high and low accept fields in details for greater 
discrimination between cases, and addition of a module for automatically 
generating the state required reports. These suggested improvements are 
summarized in the recommendations chapter which follows. 

Positive Features of the DSS Model 

One of the major questions which was being tested in the pilot phase of 
the project was whether or not the DSS model could be successfully 
implemented in a variety of different settings. The positive feedback fi-om the 
pilot test sites affirming the perceived value of the system provided convincing 
evidence that the model was indeed useable in very different ROC/Ps. Eight 
of the ten sites which had produced reports using the system responded that 
they could benefit from implementing the system as it was. Several were 
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enthusiastic in their evaluation of the ease with which the system could be 
understood and used 

■ In general we found the documentation useful and clear* The 
format was appropriate and easy to follow* 

B The manual was simple to read and right to the point* It left out 
the jargon typical of software manuals* 

■ I found the DSS software weU constructed, weU thought out, and 
hard to screw up. With very httle formal inservice I was able to 
figure it out. 

■ The DSS is a great software program. The manual is excellent. 
It has gireat potential as a management tool. 

■ The usability of the system is excellent if the data can be 
efficiently entered 



Of the many featiures of the program, one of the favorites among 
managers was the user interface. Managers liked the fact that they could 
interact with the data and defme so many of the parameters on which the data 
were being analysed For some it was an eye-opening experience to be in a 
position to manipulate their course data* 

■ We liked the general approach: a system built on a microcomputer 
with an easy to use end-user interface* We rarely referred to the 
documentation for operational issues; it vtiaS clear how to enter 
and modify data* 

■ We have had a system for course evaluation which does much of 
the same things as the DSS. The DSS LdS a couple of advantages 
over our system. It is nice to be able lo work with data on a PC. 
The DSS is much more flexible in that we can easily define new 
details and easily modify the weights* 
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■ Being able to interact with data on the computer has helped me 
realize the value of seeing the data and being able to work with 
it. 

■ It was a good idea to provide managers with the options for 
changing the parameters of data reporting. I especially like the 
ability to weight and add more details and to change the 
parameters to imderstand the relationships between details. 

■ I think the DSS is going to be a terrific tool It will be very useful 
to me. I like the way I can build additional details into the 
analysis. Fm pleased that you guys could develop something like 
this in such a short time. It takes into consideration the 
differences between different ROC/Ps and yet provides a generic 
tool that all can use. 



In addition to the feature of user interface, the reviewers appreciated the 
system for its power in supporting the decision making process, particularly the 
fact that it could help them pull together information from a variety of sources. 

■ The DSS reports provide a good guide for course planning 
decisions. 

■ The DSS has the potential of putting things into perspective. It 
also helps avoid biases and subjectivity in decision making. 

■ This is a potentially very valuable system. If the DSS were a high 
priority, it would improve the position of all the managers in this 
ROP. We would work a lot smarter. We now spend a lot of time 
doing dumb stuff. 

■ In the past our databases have been in different places, it was 
impossible to make the kind of comparisons that the DSS does. 
With the DSS we will be able to follow up problem areas instantly 
instead of waiting until all the data is in from different sources. 
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One of the assumptions behind the choice to begin the development of 
an MIS with a decision support system was that a DSS would drive the data 
collection process Mdth positive and intrinsic motivators. The research 
Uterature suggested that this would be the case, but we had no way of knowing 
how soon to expect that this would happen, or if it would even be evidenced 
within the short time available for pilot testing of the first prototype of the 
DSS. The feedback &om three or four of the cases confirmed that a decision 
support approach to system development can and does generate intrinsic 
motivation for data collection and standard setting. 

■ The DSS helps to focus the priorities and standards and helps to 
drive data collection. Just talking about pilot testing of the DSS 
has already changed the thinking a bit in this ROP. 

■ I would like to have a year to work on the implementation of the 
DSS. That way I could plan ahead and collect section specific data 
to include in the report at the end of the year. 

■ I will be a much better user of the DSS in the futinre because I 
will be able to use more details and experiment imtil I get the 
weights right. I will also be able to develop information to add to 
the system. 

R The discussions of the DSS even prior to pilot implementation 
have emphasized the weakness of our current data to support 
decisions and prompted a decision to drop our current data 
collection system in favor of something better. 

B The DSS could save us money by allowing us to look at costs in 
connection with quality and improve both the cost efficiency and 
the quality together. 

B One of the positive features of the DSS is that it gets us thinking 
about standard setting. 
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These quotes from system reviewers provide strong afiBrmation that the 
DSS model can not only be implemented in a variety of different settings, but 
that it has the potential of revolutionizing the collection and use of data by the 
managers of ROC/Ps. Instead of simply collecting data because it is required 
for a report, ROC/Ps moII be motivated to find more thorough and accurate 
ways to measure and dociunent the positive effects of iheir programs and 
support management decisions. 

Findings III: Uses of the System 

To determine their views about the system's uses, reviewers were asked 
to volunteer uses for the system and its reports and then were asked to 
comment on a list of possible uses suggested by the developers. The findings 
resulting from this exercise are sxmunarized in the following sections. 

Uses Volimteered bv System Reviewers at Pilot Sites 
System reviewers at the pilot sites volunteered a list of uses similar too 
that suggested by the developers. The major use volunteered was to provide 
a base of program evaluation data from which to draw in making 
decisions about courses. This process was described in fotn* sub*steps: 

a. analyse cost benefit ratios and other measures of course quality 

b. set target goals for improvement in various measurable areas 

c. evaluate overall course quality with combined measures 

d. conduct longitudinal evaluation of changes 
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As expected^ the decisions identified as being informed or supported by the 
system included those in the areas of course development, budget planning, 
scheduling, and the mix of course offerings. The reports were seen as useful 
for informing and/or motivating advisory committees, boards, school districts, 
counselors, and instructors. ROC/P leadership also viewed the ^stem as a 
possible alternative to the enacted requirements for: 
a. course approval 

a. biennial coiu^ quality review 

b. compUance review. 

In addition, several unique uses for the system were suggested: 

■ It seems hke this system could be very useful for a comprehensive 
high school. But since they do not have the same strict 
requirements for accoimtability they may not see the need of it as 
much as we might in ROPs. 

■ The DSS would have more utility at the college or junior college 
level where the student cUentele is more mature and stable. 

■ The DSS would eventually provide a great database for research. 

■ Tlie DSS reports will assist with the appUcation for funding. 

■ The DSS could be a great PR tool. 

Responses to Svstem Uses Suggested bv Researchers 
7rhe researchers had suggested a list of seven possible uses for the 
system (see the last page of the debriefing interview form in Appendix B) and 
asked the pilot sites to comment on them. Responses on the seven possible 
uses varied from site to site. These responses are particularly instructive as to 
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positive prospects and problems to expect in implementation and use of the 
system in the futm^e. In the following sections responses to each suggested use 
of the system are briefly discussed 



!• Could the DSS be used for assisting managers in 
making decisions to retain, suspend, or drop 
courses/sections? 

The response to this question of ROC/P managers who had reviewed the 
DSS software was overwhelmingly positive. Several of them emphasized their 
agreement with statements such as: 

■ Most definitely! This is the msgor reason for having the system. 
The use of the DSS to support decisions is clearly one of its strongest selling 
points for ROC/P managers. 

2. Could the DSS be used to justify course/section 
decisions to superintendents or ROP boards? 

The majority of pilot sites responded positively to the use of the DSS to 

justify local management decision, but responses ranged from very positive to 

cautiously qualified On the extreme positive end one manager stated that this 

wa/i the most important use of the system. In another site the manager 

interviewed explained that such a use would not be the most politically astute 

in situations where the superintendent has a great deal of decision making 

power. One ROC/P director shed some Ught on the possible differences 

between ROC/Ps with different governance structures: 
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■ Whether or not the DSS could be used to justify ROP 
management decisions would depend a lot on the political 
processes in the particular ROP office, JPA operations may have 
more leeway in decision making - the CEO has more control than 
the CEO in most county operations. It also depends on whether 
the ROP contracts everything out to districts or controls the 
operation of coiwses themselves. It also depends a lot on whether 
the superintendent was appointed or elected Elected officials 
tend to wield a lot more power in the decision-making process. 

In other pilot sites managers explained that because of their managerial 

autonomy, such a use of the system would not be appUcable. The importance 

of using the DSS reports to defend local management decisions to other 

poUtical entities may or may not be seen as a positive option to ROC/P 

managers, depending on their situation. 



3. Could the DSS provide feedback to mid-level managers 
to help focus interventions and inservices? 

All the responding sites saw the feedback to help focus interventions and 

inservices as a benefit of the DSS. The DSS is designed to support decisions 

related to the targeted improvement of coinrse quality. This feature can 

certainly be used as a major selling point for the system to ROC/P managers. 



4. Could the DSS provide feedback to instructors on the 
relative quality of their class sections? 

Response to the possibiUty of using the DSS reports as direct feedback 
to instructors was mixed Two managers were quite positive about keeping 
instructors informed on their relative st atus and on the measures by which the 
quahty of their courses were being determined, especially on those measures 
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over which the instructors had some control such as student attendance and 
student reviews of instructional quality. Most managers responded that they 
would use the DSS reports only on a selective basis with individual instructors. 
In one site the manager explained that the instructors had voted not to 
pubUcize any comparative data regarding their teaching. Clearly the use of the 
DSS reports as feedback to instructors would have to be approached with 
sensitivity on a site by site basis, depending on the needs and attitudes of those 
involved. 



5. Could the DSS provide report data to share among the 
other ROC/Ps in the CAROC/P association? 

On the possibility of using the DSS to generate information to share 

between ROPs, most of the responding sites saw it as a desirable goal, but one 

which would be difficult to achieve given the differences in data defmition 

between ROPs. One site suggested that ROC/Ps could share via FAX machine 

on a number of variables they already had in common: 

■ The DSS could provide sharing between ROC/Ps on things like 
placement rates for different courses, connections with higher 
education, and rates of continuing students. 

Another site was much less sure about the benefits of sharing data between 

sites. The concern of this manager had to do with misuse of aggregated data: 

■ Sharing data between ROPs is not something I can see you would 
want to do. Aggregations can screw up data quite badly. While 
compiled reports may be interesting, they are too easily misused. 
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The msgority of pilot site managers suggested that data shariiig would only be 
possible after the ROC/Ps could agree to conunon data definitions and 
collection protocols. The use of the DSS for sharing of common data among the 
ROC/Ps is an option which needs to be explored more carefully. A study of the 
common data elements which would be useful to know about other ROC/Ps 
should be included in the next phase of DSS development, so this use of the 
system could be facihtated. 

6. Could the DSS be used as a database from which state 
reports are generated? 

Responses on the use of the DSS as a system for generating state reports 
were mixed Of the ten who responded, six saw this usage as a good possibihty 
in the future once the DSS was in place in a number of sites and some thought 
had been given to common data elements which may be of interest to the state. 
Another three others said it would only be possible if consensus could be 
reached on state level data definitions. One pilot site manager answered with 
a flat "no". 

7. Could the DSS be used to replace or supplement the 
course compliance review process? 

In response to the question of how the DSS would relate to course 
comphance reviews foin: managers saw some possibihties for the DSS 
eventually replacing the CCR One ROP manager suggested that using the 
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DSS for state reporting should be a low priority compared to the use of the 

DSS for self evaluation and course quality review. 

■ If I were the state Fd get behind this system one thousand 
percent. The state shoidd mandate the use of the DSS and 
reduce the compliance reviews. They could simply send an 
auditor to check that careful self-evaluation is being done. 

Four other managers who responded to this question were favorable to the use 

of the DSS as a supplement to the CCR process. In the next round of 

refinements of the DSS, careful thought needs to be given to the elements of 

the CCR which can be defined as details to be ijicluded in the decision support 

^stem. As the use of the DSS for local decision support is expanded and 

refined, it may be found that many of the same elements included in the CCR 

are being covered by the use of the DSS. An eventual goal for the DSS should 

be to incorporate the elements of the CCR into the state reporting linkages. 



Summary 

The project was successful in identifying local ROC/P coiurse-level 
management information needs and developing a proto-tj^e decision support 
system to address many of these needs. The pilot test showed the system to 
be adaptable to a range of ROC/P settings representing different organizational 
structures, sizes, and technical capacities. A nmnber of important uses were 
identified by the designers and by those who reviewed the system. Pilot site 
users and other ROC/P managers have expressed enthusiastic interest in the 
development and implementation of a production version of the model system. 
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Much remains to be done before the MIS model can become the 
backbone of local program management and statewide analysis that it deserves 
to be. The existing software is clearly developmental in character and will need 
additional work before it is ready for production use. Production ready software 
camot be completed until local ROC/P managers have acquired considerably 
more experience with the decision support concept and have used it to anal3rze 
a much broader array of program details. Staff development and training work 
needs to be undertaken to enable local managers with limited computer 
expertise to become comfortable with the MIS approach incorporated into the 
existing software. Major development work needs to be done to link local MIS 
usage to the data and accountability needs of state level administrators and 
policy makers. Recommendations regarding the crucial development and 
implementation needs are included in the executive summary at the beginning 
of this report. 
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Detail Definition Worksheet 

For Specifying Quality Measures and Performance Indicators 



D«Uil 


Title 


Kind, 
Type 


Minimum, 
Low Accept 


High Accept 


Default Value, 

f A 9 nv 9 A A. 

Direction, Wei^t 


$ADA 


Direct Cost Per ADA 


A 


$500.00 


$10,000.00 


$3,000.00 


D— cription 


D 


$1«800.00 


$3,500.00 


< (lower better) J 


AO dirMt co&U for a Motion (Salary, BanofiU, SuppliM, Facility, 
Equipment) divided by the ADA generated by tbe eeetion 


35 1 


CERT 


Percent of Certifications 


A 


0% 


100% 


65% 1 


DeecHptioa 


P 


50% 


80% 


> (higher better) | 


Total number of students oertifloated by the end of th 
1 for a seesion divided by the total number attending 


e allotted time 


30 1 


POSX 


Percent of Positive Exits 


E 


0% 


100% 


50% 


Dceoription 


P 


40% 


70% 


> (higher better) 


Total number of students in military, farther study, or Jobs within 3 
months of the end of the session divided by aU attendees 


35 

— — 
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I. PILOT SITE DESCRIPTION 
Site Identifiers 

Name: 

Location: 

Administrator: 



Administrative Structure fcoiiect organizational 

Chart) 

Type: CO JPA SD Other 

□ □ □ 



Numbers of: 

Administrative Staff: 
Classified Office Staff: 
Instructors: 

Relative Size 

Longest Distance from ROP to site: 

Numbers of: 

Districts: 

High Schools: 

Instruction Sites: 

Courses Offered: 

Sections: 

Students Served: 

Annual ADA: 

Base Revenue Limit: 



Service Area Characteristics 

Population Density: Rural Suburban Urban 
Population Ethnicity: 
Population Growth Rate: 
Job Market Diversity: 
Job Market Demand: 

Extent of Transportation Provided to Students: 
Other Factors: 

Technical Capabilities in ROP 01 

Personal ComHiter^ J 

Types: 

Count: 

Uses: 

Modem: 
History: 

Person Contributing Data: 
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III. DETAIL DEFINITIONS 



1 DETAIL CODE 
1 QUESTION 


$ADA 


CERT 


POSX 






1 How Is this detail 
1 defined? 












What formula Is used? 












Where was data for Values 
found? 












How were the values data 
collected? 












i What format were the data 

n 












1 Who collected and 
1 calculated values? 












1 Were values available on 
i the section level? 












1 How were the parameters 
1 defined? 












m Wh^it HoPQ thiQ Hpfail 

1 measure? 












1 How Important Is this 
1 detail? Should It be 
1 required of all ROC/Ps? 












1 How easy was this detail 
B to Interpret? 
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IV. ANALYSIS REPORTS GENERATED (collect i.test 3 reports) 

Analysis Definitions Impact of A nalysis Reports 

Who defined these analysis reports? 

Decided which details to Include? 
Set the weights, high and low accepts? 

Are these First, Second, or Third generation reports? 

Why changed? 

Report Exposure 

Is It right yet? 

Who has seen these reports? 

What was the context? Internal External Were any management decisions Influenced by the ref 



Were parameters evaluated or changed? 



In what ways changed? 



What were the initial reactions? 



Confirms what we know 

Surprised 

Puzzled 

Confused 



If so, in what ways? 



If not. why not? 



What discussions took place? 
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Tenor of discussion? 
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A Management Information System for 
California's Regional Occupational Centers and Programs 



Mid-Project Executive Summary 

A RMMTch and Dtrtloprntnt Prrc^ Conductod oa BduOf oTCAROC/P hy tbt 
difomU Eduotfonml JUmmh Cooptr^ve at th« Unhrtwity of Criifon^ 



RATIONALE 

A Management Information System (MIS) 
is a ^stematic, planned way of xising information 
to make good management decisions, A well 
designed MIS for California's Regional 
Occupational Programs and Centers (ROPs) will 
assist the local managers in effective decision 
making, and also provide the CAROC/P 
organization with a vehicle for collecting a 
statewide sample of miiformly defined data- 

ROPs have found themselves in an isolated, 
reactive stance toward legislative changes. The 
CAROC/P organization would like to take a more 
proactive stance toward the governance of the 
ROPs- With the increased standardization of data 
resulting from a commonly shared MIS, the 
statewide organization would significantly 
strengthen its voice in state level pohcy decisions 
regarding the operation of ROPs, 

TIME LINE 

The MIS development project is scheduled 
for three phases: (1) MIS development, (2) MIS 
piloting, and (3) Data analysis/reporting. The first 
phase involves a literature review, visitation of 10- 
12 ROC /P sites, work sessions with ROP managers 
and other field representatives to outline MIS data 
elements and relationships, selection of several 
pilot sites, and development of the MIS and 
related materials. The second phase includes such 
tasks as orienting the pilot sites to the MIS, 
collecting data, analyzing and reporting the data to 
site managers, and evaluating the MIS for its 
contribution to management decision making. 
The third phase consists of collecting and 
analyzing the common data generated at the pilot 
sites, getting valuative feedback from participating 
individuals, making revisions in the MIS, and 
generating recommendations and reports. In all 
phases of the research and development effort 
input from field practitioners will be solicited. 



The fiTiftl outcome of the study will be a 
very basic working computerized decision support 
model, not a fixiisbed conmiercial product. The 
report with recommendations should provide the 
basis for the development of a more 
comprehensive commercially viable MIS which 
could used by ROPs across the state. 

SCOPE 

The MIS xmder develop nent is limited in 
its scope to the evaluation of course related data, 
primarily addressing the questions of course 
quality. It will be extremely useful to managers in 
establishing valuative criteria, pinpointing 
problems, determining which corrective measures 
need to be taken, and deciding when to suspend or 
terminate courses. 

The early phases of the research found that 
many of the ROPs have reasonably sophisticated 
data collection systems in place, so the proposed 
system does not attempt to replicate this. The 
proposed MIS is a top-end analysis and reporting 
system rather than a data collection systenL The 
piloting of a such a top-end MIS serves several 
purposes: 

1. It will demonstrate the extent to which 
ROP managers will make use of data in decision 
mpking — illuminate the value of a full MIS to 
managers. 

2. It will demonstrate the potential for state 
wide sampling of uniform analysis data — suggest 
directions to take in the design of a state wide 
system. 

3. It will allow the ROPs to assert the 
measures by which they want to be held 
accoimtable rather than waiting for the state to 
tell them - take a proactive stance toward the 
current interest of the state legislature in 
accountability. 
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List of data needs of ROC/P managers from CEO forum 
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CEO Brainstorming Session Summary 

• 



Setting for the CEO work session 



The annual CEO Forum was held in Palo Alto, CA. September 21-22, 
1990. An hour-long work session was conducted with the ROC/P CEOs 
present. The participants were divided into 5 groups and asked to identify and 
establish priorities on the issues and management decisions related to each of 
the stages in a course life cycle, (1) goal clarification, (2) needs assessment, (3) 
development, (4) operation, and (5) evaluation. The session served to raise the 
awareness of the CEO's of the MIS project, bring about a higher degree of 
buy-in than could have been accomplished through a monologue presentation, 
and verify the key issues thought to be central to the function of an MIS. An 
outline of the issues generated by each group is provided in the following five 
sections. 



Mission and Functions of ROPs 

The mission of ROC/Ps is to prepare youth and adults for entry level 
jobs; provide advanced training and up-grading skillfl to ciuxently employed 
individuals, and to provide retraining. The statutes restrict the lower age lixnit 
of ROP recruits to high school students above the age of 16 or a junior in 
standing. Special needs individuals are not excluded, but are not particularly 
targeted for service by ROPs. 

Accountability 

ROPs are seen to be accountable in three ways: 

1. Legislative 

2. Regulatory 

3. Socially 




Group 1 - Goal Glarification 
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Group 2 ^ Nee^^^^^ 



Priority Considerations and Questions 

The group working on needs assessment outlined the following 
priority issues (numbered and bold) to consider when asking whether or not 
to offer a new course/program. The questions following each issue were 
generated by the researcher. 



1. Labor market 

a. Is there sufficient demand for individuals trained in the way we 
could realistically train the students whom we serve? 

b. How many job openings of what type can be expected for every 
traimng period? 

c. How willing are employees to hire our trainees or work with us 
on commimity classroom arrangements.? 

d. Will this labor market last long enough to warrant the efforts 
of planning and starting a program? 

2« Curriculum 

a. Is curriculum available for the proposed coiffse/program? 

b. What time and labor efforts will be needed to develop 
curriculum which is appropriate to the labor needs and will 
meet state standards? 

3« Monetary 

a. Can we afford to add a new program at this time? 

b. What are the likely start-up costs of the proposed course 
(curriculum development, materials, equipment)? 

c. What are the on-going operations costs of such a program 
(instructor, facilities, suppUes)? 

d. What soiu*ces of income other than ADA are available to offset 
the costs of this course? 

e. Can enough ADA income be generated to balance the costs of 
the cowrse? 

f. Will increased wage earning power of trainees justify the cost of 
training? 

g. Will the likely placement rate of trainees jxistify training costs? 
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4. Student interest and availability 

a. Is student interest high enou^ to expect to fill the class? 

b. How many students can be ejected to take the course? 

c. What advertising efforts will be needed to assure adequate 
enrollment? 

d What will the advertising cost? 

e. Which form of advertising is most effective? 

f. What is the minimum munber of students needed to break 
even on cost and income? 

g. What is the maximum number of students the coiu^se can take 
given the facilities, equipment, materials, and instructor(s)? 

h. Are students physically near to the training site(s)? 

i. Is transportation necessary, feasible? 

5. Articulation/Non-duplication 

a. Will this course/program provide an appropriate step for 
students who plan to continue training in this field? 

b. Will this course articulate with area colleges and universities? 

c. Is this course an unnecessary dupUcation of similar courses 
offered elsewhere? 

d Where else could students get the same training? 
e. Is the course filling a real need? 

f What related educational programs in the region are likely to 

be impacted by the addition of this course? In what ways? 
g. Can we Uve with this impact? 

6. Internal ROP needs and constraints 

a. Do we have the necessaiy resources to begin and sustain this 
coiirse (facilities, instructors, equipment, materials)? 

b. Do we have the political support to begin this course? 

c. Can we add to our enrollment without going significantly over 
our cap? 

d What effects will the addition of this course have on other 

courses/programs? 
e. What effects will the course have on the administrative work 

load? 

f- What steps are required to bring a new course fi-om concept to 
complete integration with the whole program? 

g. How long will this process take given the available himian and 
other resources? 

h. How much will it cost? 
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7. External restrictions and constraints 

fiL Will this coxirse meet government approval standards/ 

b. Are there labor mnon standards which must be met for such a 
program? Can they be met? 

c. Are the expectations of potential employers within reasonable 

reach for this course? 
d If not, can these expectations be modified to match realistic 

competency outcomes? 
e. Can our facihties meet state bmlding safety codes? 



Group 3 - Course Development 



Areas of Consideration 

State frameworks (curriculum) 

Class size 

Facilities 

Equipment 

Costs and monetary effects 
Credentialed Teacher 
Advisory Committee 
Job Skills Analysis 
Competencies 
Articulation 2+2 

Curriculum comparisons with other ROC/Ps 
Type - Class only, Community class, Co-operative 

Sequence of major tasks 

1. Form an advisory committee with the following functions 

a Job task analysis 

b. Job titles listing 

c. Competencies development and verification 
d Articulation with higher programs 

e. Community classrooms and CVE coordination 

f. Equipment and facihty recommendation 

2. Match fi-ameworks with quaUty indicators and competencies 

3. Review other model programs 

4. Evaluate Costs 

5. Assure 2+2 Articulation 
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roup 4 - Course Operation and Mcmitoring^ 



Seven top priority components 

The following components were rank-ordered by the CEOs in group 4 on 
degree of importance in the task of managing a given course on a day-to-day 
basis. Questions and commentaxy were generated by the researcher. 

!• Attendance 

a. Do we have enough students to warrant offering this coiu^? 

b. Are enough students attending to generate the ADA needed to 
fund the offering of this course? 

c. Does the level of attendance point to quality concerns in the 
curriculum or its delivery which need to be addressed? 

2. Costs 

a. What is the budgeted cost for this course? 

b. How much ADA would have to be generated to cover the cost of 
offering this coxurse? 

c. How much of the budgeted costs for this course are fixed costs 
and how many are flexible costs? 

d. Where could cost cutting be applied to this course? 

e. Which costs will be covered by revenues other than ADA? 



3. Effects of Scheduling 

a. What is the best time for offering a course? 

b. When will the most students be able to attend? 

c. Is transportation needed? Available? 



4. Cmriculum Relevance 

a« Have we designed the course to prepare for appropriate jobs? 

b. Is this course preparing students for available jobs? 

c. Are graduates of this course able to do the jobs well? 
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5. Quality of Instruction 

a. Is the instructor of this conroe properly qualified? 

b. If not, what needs to be done to qualify the instructor? 

c. Is the instructor competent? What needs improvement? 
dL Are students learning what they need to know for jobs? 
e. Are students happy with the qucQity of instruction? 



6. Facilities and Equipment 

a. Do the facihties and equipment used in a course meet 
govermnent standards? 

b. Are facilities and equipment appropriate to the course 
objectives - job titles toward which training is oriented? 

c. Is equipment inventoried, serviced, maintained properly? 

d. When is it appropriate to retire old equipment and purchase 
new? 

e. What are the appropriate equipment depreciation schedules to 
set? Are they clearly documented? Are depreciation costs 
accounted for in the total cost of a coturse? 



7. Student Follow-up Data 

a. Do students like this course when they are done? 

b. Have students met their entrance goals at exit firom course? 

c. Why do students leave this course? 

d. What can students who complete this course do? 

e. What do students do when they complete this course? 

f. Which course objectives are best met by this course? Skills 
attainment, placement, avoidance of dropout, personal 
satisfaction of students? 
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II Group 5 - Course EvaJ^^ ^ 

Outline of concerns and issues identified by CEOs 

1, Completers/Leavers (needs to be redefined) 

fiL Enrollment 

b. student needs and interests 

c. accessibility 

d teacher availability 

e. graduation requirements 

1 counselors 

2, Cost Effectiveness 

3, Job Market Analysis 

a^ Wages 
b- Futm'e 
c. Placement 

4. Advisory Committee 

5. Public Perception 

a« Termination 
b. Observation 

6. Curriculum Standards - Statewide 

fiL Common competencies 

b. Standardized testing (competency based) 
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Types and Examples of Information Needed by Top Managers 
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Types and Examples of Informatioii Needed by Top Managers 



Seven types of information are frequently needed by top-managers in an 
organization. This document lists definitions and examples of these 
information types reflecting the ROC/Ps in California. 

Comfort informcUion: keeps managers informed about current situations or 
achievement levels; allows the individual to know that performance is on 
track and in line with general expectations in an area of interest. 

Examples: 

Current enrollments by section 

Actual ADA generated compared to expected ADA 

Number of completers 

Status information: also called progress information; keeps managers 
abreast of current problem and crisis as well as reporting advances to take 
advantage of opportunities that may disappear if not acted upon. 

Examples: 

Number of absences or drop outs 

Status of advisory committee meetings 

Progress on new course developments 

Progress of other agencies on developing similar courses 

Warning information: signals that changes are occurring, either in the form 
of emerging opportunities or as omens of trouble ahead that will affect the 
success of the firm, its products or services, or its long-term viabihty. 

Examples: 

Shifts in the labor market 

Legislative changes in funding or reporting requirements 
Unusually high or low completion or placement rates 
Rapid increase in drop outs from a particular course 
Increase in student interest in a course 



Planning information: descriptions of major developments and programs 
due to begin in the future; includes assumptions on which plans are based or 
anticipated developments essential for the realization of the established 
plans. 
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Types and Examples of Information Needed by Top Managers (cont.) 



Examples: 

Labor market growth or decline 
Entry of new labor market sectors 
Entry of competitor - other training agency 
Availability of qualified iEstructors 

Internal operations information: key indicators of how the organization or 
individuals are performing; useful for reporting the overall health of an 
organization, subsidiary, division, cr product. Areas in which actual 
performance does not match expectations are reported as exceptions. 

Examples: 

Biennial course quaUty review 
Student satisfaction survey information 
Follow-up information on placements 
Revenue/expense reports 

External intelligence: information, gossip, and opinions about activities in 
the environment of an organii^ation; includes a broad range of areas such as 
competitor and industry changes, financial market movement, and poUtical- 
economic fluctuations or expected shifts. 

Examples: 

Industry demands for new types of trained workers 
Expert projections of economic behavior in next 6 months 
Talk of actiors in hi^ school, community college, and adult ed. 
The fall-out firom legislation in vocational education 
Student interest survey results - availabihty of students 

Externally distributed information: information the chief executive wishes 
to review before its release to stockholders or distribute to the news media. 

Examples: 

Semester reports of successful completions 
Acctunulated contributions firom business and industry 
Details of newly developed student services program 
Reports of successful placement of workers into local business 

Reference 

Senn, JA. (1987). Information systems in management (3rd ed), California: 
Wadsworth Publishing Company, p. 32-34 
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Flowchart of course life cycle management decisions 
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Flowchart of Management Decisions 
in the Life Cycle of a Course 



Planning Phase 


1 SOURCES OF 

TXTfTITnTJXT A T 

INTERNAL 
SUPPORT 
DATA 


MANAGEMENT 


SOURCES OF 
EXTERNAL 

" f ^ * 1 J 1 111 T il 1 J 1 

SUPPORT 
DATA 




Is there sufficient local political support 
to launch the proposed course? 


ROC/P Board and 
advisory committee 
input 


Personnel cost 
records 


Is there sufficient student interest in the 
proposed course to cover the cost of 
hiring an instructor? 


Student interest 
surveys in high 
schools 


Past placement rates 
for similar courses 


Is the lahor market demand enough to 
provide johs for course graduates? 


EDD projections, 
Advisory committees 


V/OunXS OI CUlupiCbCia 


market needs in different fields? 


EDD statistics 




served by existing training programs in 
1 the region? 


Survey of programs 
offered in region 
(future AEI MIS) 




What equipment/supplies are essential 
to the success of this course? 


Advisory committee 
input on job skills 
analysis and 
equipment needs 


Personnel, Inventory, 
Facilities, and 
Transportation data 


Do we have access tc the basic 
components needed to run the course? 


Possible new hire 
instructors), new or 
donated equipment 


Projected costs for 
instructor, supplies, 
equipment, feicilities 


Do we have the funds needed to offer 
the proposed course? What is a 
reasonable investment to make in 
establishing this course? What are the 
likely benefits? (social, financial, 
political) 


Possible sources of 
funding other than 
ADA allocation, 
records of similar 
courses elsewhere. 


Expected life of the 
course. Past practice. 


How should one-time startup costs be 
amortized? 


Record of similar 
courses elsewhere 
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Flowchart of Management Decisions 
in the Life Cycle of a Course 



Course Development Phase 


SOURCES OF 
INTERNAL | 
SUPPORT 
1 DATA 1 


MANAGEMENT 
DECISIONS 


SOURCES OF 

U'VI'L'TJXT AT 1 

SUPPORT 
DATA 1 




For which jobs will this course prepare 
its graduates? 


UlCuOi uccupauonai 
Tities, OES-CBEDS- 
GIF crosswalks 




What are the essential skills and 
knowledge needed for efEiective entiy 
level employment? 


Advisory committee 
inpuvy empioyer 
survey 




Which skills and knowledge would 
qualify a student as job readyr 


Advisoiy committee 
frameworks/guides 


Course lists of 

achievable 

competencies 


Do the competencies laentinea in course 
outlines match essential job skills? 


•Inb skills analvsis. 
OES-CIP crosswalk 


Student demographic 
statistics and 
learning patterns 


How many hours should it take to train 
the type of students we serve to a state 
of job readiness? 


Information on 
similar coxirses 
offered elsewhere 


Course description 
database 


How well does this course articulate 
with other educational opportunities? 


Related college 
course prerequisites 


Facility size, available 
equipment, cost and 
training of instructor 


How many students will the course 
serve? ideally? maximally, minimally? 






Can the course meet state guidelines for 
approval? 


VE 77 forms and 
process 


Database of 
sponsorship 
agreements 


How can regional businesses be involved 
in the support of the students in this 
course? 


Chamber of 
commerce, labor and 
emplojrment offices 



ERIC 



CAROC/P MIS 



Page 16 

154 



CERC @ UCR 



Flowchart of Management Decisions 
in the life Cycle of a Course 





Implementation Phase 




SOURCES OF 
INTERNAL 
SUPPORT 
DATA 


MANAGEMENT 
DECISIONS 


SOURCES OF 
EXTERNAL 
SUPPORT 
DATA 


Survey of students* 
methods of discovery 
about the course 


What is the most cost effective method 
of attracting good students? Which 
types of advertizing attract which 
students? 




Survey of student 
intent/expectations 
at outset of course 


How can our students best be served by 
the course? What are studeat 
expectations from the course? 


Student interest 
8Uiv^ in high 
schools 


Past placement rates 
for similar courses, 
enrollment, student 
intent survey data 


To what extent is this course likely to 
contribute to labor supply? 


COICC data 
collection forms and 
procedures 


Admissions database 
of student 
demographic data 


Are the students who are attracted to 
this course representative of the normal 
population? Might our advertizing need 
to change to attract a more normal 
1 group? 


Ciomparative high 
school student body 
descriptive statistics, 
local census data 
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Flowchart of Management Decisions 
in the Life Cyde of a Course 



Operation Phase 


SOURCES OF 
INTERNAL 
SUPPORT 
DATA 


MANAGEMENT 1 
DECISIONS 1 


SOURCES OF 
EXTERNAL 
SUPPORT 
1 DATA 


Enrollment or 
attendance database, 
course descriptive 
data on ideal, min 
and max enrollments 


Does the enrollment warrant 
continuation of the course offering? 
addition of sections? 




Budget, ideal and 
actual enrollments, 
attendance patterns 


Does the ADA revenue generated cover 
most of the costs of the course. 




Attendance records 
and reports 


Which students have chronic attendance 
problems? How can they be encouraged 
to attend with regularity? 




Supervisor evaluation 
spending records 
attendance patterns 


How well is the instructor of this course 
doing? 




Observations, course 
outlines, completion 
rates 


Does the curriculum taught in the 
course match the objectives in the course 
description? 




Course outlines, 
competency 
achievement rates 


Are students learning the intended 
competencies in the time allotted? 


CVE supervisor 
evaluations of 
student skills 
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Flowchart of Management Decisions 
in the Life Cyde of a Course 



Evaluation Phase 



SOURCES OF 
INTERNAL 
SUPPORT 
DATA 




MANAGEMENT 
DECISIONS 



SOURCES OF 
EXTERNAL 
SUPPORT 
DATA 



Exit survey compared 
with entrance survey, 
follow-up survey 

Course dropout rate, 
Dropout follow-up 



Achievement, com- 
pletion, & dropout 
rates, course outline 

Student intent and 
placement rates 

Entrance survey of 
wages and follow-up 
survey of wages 

Follow-up data linked 
to intake survey data 
by individual student 

Student demographic 
data, completion and 
placement rates 

Compiled and 
anal}aed data from 
many sources 

lists of employers of 
course graduates, 
employer surveys 



Is the course meeting the needs of the 
students who attend it? 



How can dropout rates be minimized? 
What are the primaiy causes of 
dropping? 

Is the curriculum appropriately difficult 
and the instruction corxBctly paced for 
the students served? 



Does the course provide sufficient links 
with employment for graduates seeking 
work after completion? 



To what extent does the training 
provided in the course increase the 
earning power of graduates? 



How does the course affect the career 
goals and choices of the students who 
attend and complete the competencies? 



What do the state and federal 
governments need to know about the 
effects of the course? 



How does the overall quality of this 
course compare with other courses 
offered by the ROC/P? 



Who are the primary employers of our 
graduates? How can we better serve 
them? 



Dropout rates of 
comparable group in 
high schools 

Employer satisfaction 
surveys 



EDD statistics on 
wages for various 
jobs 



State and Federal 

reporting 

requirements 



Advisoxy committee 
input 
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Graphic figure showing the domains of course quality 
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Domains of Course Quality 
Measurement and Analysis 



Continued 

related 

training 



retention 
graduation 



Transfering 
and reporting 
of student 
educational and 
demographic data 
between 
institutions 



Curriculum 
Articulation 



Transport 
Scheduling 
Facilities 
Equipment 



Equity and 
Equality of 
opportunity 



Success 
on the job 
and in life 



Competency 
achievement 



Instructional 
quality 



Appropriate 
assessment, 
placement, 
counseling 



Cost analyses: 

Efficiency 
Effectiveness 
Benefits 



Placement 
in related 
employment 
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Table of information needs in the domains of quality 
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Course Quality Measxirement Variables 



Source of Data Illustrative Measurement Variables 



Category of Analysis 


Variable Factors 


Base Factor 


ROP ooune instructors & managers 


Cost efTectiveness 


cost per certifkatiGnt placement, etc 


mean cost per * 


Cost eflideiKy 


course costs, cost per ADA, % comcred 


means, 90%-110% 


Service to special needs students 


services available, % special students 


standards, norm distrib 


Compliance with state regulations 


forms (VE80, VE77, J780) 


standard data procedures 


StudenU A, graduates of ROP 






Impact on career choice and success 


% placemenu in related work or stady 


total enrolled in course 


StudenU ROP 


Competency achievement 


% certified by course end 


total enrolled in course 


Instructional quality 


student and supervisor ratings 


rating scales 


Employers (both actual and 
potential) 


Labor market demand 


trends, predictions, local surveys 


past predictions, time 


Resource sharing agreements 


$ value, counts, rating of qxiality 


rating scale 


Emplo>rers ROP 


Equipment & facility quality 


rated by instructor and advisory group 


standards, rating scale 


Employers -f Students 


Client satisfaction level 


student and employer rating surrey 


I rating scale 


Employers ROP •«- Students 


Curriculum relevance 


rating survey to different groups 


rating scales 


Other schools 


Student data transfer and reporting 


rating of efllciency and accuracy 


rating scale 


Resource sharing agreements 


rating of political/economic advantage 


rating scale 


Other schools •¥ ROP 


Scheduling and transport 


costs, efficiency, counU of students 


means 


Other schools -f Students 


Attraction and retention 


interest, retention rates, graduations 


H.S. norms, means 


Other schools •«- ROP -f StudenU 


Curriculum articulation 


2+2 agreements, dbase, student daU 


standards 



BEST COPV AVAILABLE 
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Review of existing information systems 




CAROC/P MIS Page 24 CERC @ UCR 

162 



Review of Two Existing Information Systems 



Two different existing information systems were reviewed during 
Phase I of the MIS study, Solano Online's "Socrates" and San Diego County 
Office of Education's "Contracts Administration System." More than one 
third of the seventy ROC/Ps in the state use some form of the Socrates 
system. Several of the largest ROC/Ps are currently using or are in the 
development stages of programs similar to San Diego's Contracts 
Administration System. The following table provides a comparison of key 
characteristics of these two systems. 



Characteristics of Two Existing 
Information Systems in ROC/Ps 



Name 


System 


Socrates 


Development 


5 years 


12 years 


Estimated per site 
cost to install 


$400,000 


$25,000 


Hardware base 


Mainframe Computer 


Mini or 
Personal 
Computer 


Functional areas 


Admissions, Attendance, 
VE 80 Reports, 
CoxuBe Catalog, 

Contracts, 
Program Quality, 
Budgets 


Admissions, 
Attendance, 
VE80 
Reports, 


1 Size of ROP 

1 where operational 


over 3000 ADA 


600 to 2500 
ADA 
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Both 'Socrates** and **Contracts Administration System** and are 
examples of state-of-the-art information systems which have been 
specifically designed for ROC/Ps. The development approach followed in 
both cases was to anal3^e the data collection, storage and retrieval, and 
reporting practices and needs in existing ROC/Ps* The primary goal in the 
design of these systems was the effidencv of data manipxilation > Both 
systems substantially increase the efficiency with which student enrollment, 
attendance, and completion data are captiured, stored, and reported 

The impUcation for MIS development firom the analysis of these 
existing software packages in the ROC/Ps was that the best way to solve 
the data problems identified does not he in adding to or expanding the field 
of existing transaction processing systems* What is needed in ROC/Ps is 
not primarily more efficient processing of information but more 
effective use of information for management purposes. The MIS will 
function as an inexpensive generic decision support system which can linked 
to a number of different existing data collection systems (e*g* attendance, 
accoimting, and inventory). Such a system will be easily implemented in a 
variety of different sites and wiU serve to equalize even the least automated 
sites at the point of effective management strategy* 



CAROC/P MIS 



Page 26 



CERC@UCR 



Document 8 



The computer model for the ROCP-DSS software 
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The Computer Model 



The computer model was developed from the conceptualization created 
by the research team, using the weighting model forwarded by Robert R 
CarkhufiF and William A. Anthony (1981). In this conceptualization key factors 
involved in critical decisions are first identified Each factor is defined as either 
a 'positive", "middle", or "negative" scale depending on the particular value of a 
case on that factor. Particular cases (alternatives) are then collected, with their 
values on the several factors determined These scales ("+", "o", "-") are 
weighted in accordance with the individual's perception of the worth of that 
factor in the overall decision making process. Finally, the alternatives are rank 
ordered according to their overall score based on the cumulative weightings of 
their scales. 

The particular operational model was conceived on the premise that the 
unit of analysis was the course. It was recognized, however, that a single 
course could be taught at different times and locations by different instructors 
in different circumstances. Thus, it became the section that was the actual 
basic tmit of analysis for the computer model. Since sections are 
implementations of individual courses tiie system would have to expect that 
coiurses would be created first, followed by specific section(s) for each course. 

With sections in place, it would then be necessary to attach attributes to 
those sections. These attributes would cany the actual information about that 
section needed for the comparative analysis. Needless to say, attributes would 
be shared across many, if not all, sections at a given site. To accommodate this 
need a basic definition of each attribute (or detail, as it was called) became 
necessary. Specific quantitative information relative to each section (a value) 
could be attached to each of these details for each section after the details 
themselves had been defined 

A comparative analysis report could finally be produced once all values 
had been entered for all details attributed to each section. Such a report would 
need to know how to evaluate the value given to each detail for each section 
(when to rate a certain value as a "positive", "middle", or "negative" and how 
much relative weighting to give a particular detail in relationship to other 
details for any given report). Given these scoring of sections on one (or more) 
details a rank ordering of sections could be produced There would have to be 
some way to report and save the parameters from each analysis report so that 
they might be easily reproduced in fiiture runs. 

Thus, the trial system needed to be able to accommodate several 
different, yet related, data files containing information on: courses, sections, 
details (the attribute definitions), values (the actual value of a detail for a 
particular section), and analyses (the mix and weighting of different details on 
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particular sections for a given report). The system would have to allow for 
basic file maintenance functions: entry into these data files, updating 
(modifying) of existing information, and deleting of old information. In 
addition, it would have to accommodate reporting of the contents of each data 
file, and the production of the actual analysis report (the most compUcated task 
of the system). 

Development of the Trial System 

It was decided to implement this model in an MS-DOS (IBM-PC or 
compatible) platform for distribution and testing in the field This environment 
had previously been determined as the one most available to the greatest 
number of potential test sites. Dbase IV (Developer's Edition) was used as the 
development vehicle for the software written. It provided a standardized and 
easily modifiable environment for program evolution, and a nm-time module 
m akin g easy the distribution and installation of completed modules. To insure 
the TniriiTniiTTi of field problems, several programming constraints were imposed 
throughout the development: 

(1) All programming was to be done in a "modular" fashion, with 
suf&cient internal documentation to enable later programmers to 
easily comprehend this implementation. 

(2) Each module was to bear a module name, revision date and 
version number to aid in debugging and problem identification 
efforts. 

(3) Menus and full-screen prompts, with "^pull-down" assistance boxes, 
were to be used at as many points as feasible, standardizing and 
simplifying the user interface with the system. 

(4) User entry points were to be tightly edited, to prevent erroneous 
data fi:om being stored in any data file. 

The imposition of these constraints lengthened program development 
time somewhat, but proved to significantly reduce problems once the trial 
system was implemented at the test sites. 

The Data Files 

Six different Dbase IV data files are used in the CAROC/P Decision 
Support System: COURSE, SECTION, DETAIL, VALUE, ANALYS, and 
REPDAT. The COURSE file is usually the first one encountered in the system. 
It contains the basic information about each coiu^e, and a four character field 
(PROGRAM) detailing which program the course is a part of. This field may 
be used for selecting only those courses belonging to a certain program during 
the analysis reporting procedure. 
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struettur* for database: COURSE. DBF 



Fiald Fiald Nana Typa Width 

1 COURSE Numeric 8 

2 TITLE Character 30 

3 DESCRIP Character 60 

4 PROGRAM Character 4 
** Total ** 



The COURSE data file maintains two standing indices. The first, under 
the tag name of COURSEC, is just an index of the course numbers. This index 
is useful for course look-ups in the file during routine maintenance activities. 
The second, under the tag of COURSEP, is an index of the file according to 
program. This index is useful when reporting according to only certain selected 
program codes. 

The SECTION file is typically encountered next. After the \iser has 
entered one (or more) courses these courses must have one (or more) sections 
entered for each. Since values are associated with particular sections (not 
courses), it is essential that each course have at least one section. 

structure for database: SECTION. DBF 



Field Field Name Type Width 

1 COURSE Numeric 8 

2 SECTION Nvuneric 4 

3 TITLE Character 30 

4 DESCRIP Character 60 

5 LOCATION Character 4 
** Total ** 107 



Sections are unique according to a combination of the course and section 
number. This combination forms the first index for the SECTION file, a 
twelve character string combining the course and section numbers imder the 
tag SECTIONC. Like in the COURSE file, this tag is useful for locating 
particular section records during maintenance procedures. The section index 
tag, named SECTIONL, is an index of the section file according to location. 
The location code is a fom- character stni^ the user can use to represent the 
physical location (or any other useful attribute) associated with each section. 
This code can be used at analysis reporting time for selecting only certain 
locations to report on. 

Once courses and sections have been established in their respective data 
files the user next proceeds to create definitions for the details associated with 
the sections. This information is stored in the DETAIL data file. 

structure for database; DETAIL. DBF 

Field Field Name Type Width Dee 

1 DETAIL Character 4 
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2 TITLE 



3 DESCRIP 

4 KIND 



5 TYPE 



Character 
Character 
Character 
Character 



20 
60 

1 
1 

12 
12 
12 
12 
12 
1 
3 



6 MINVALUE 

7 MAXVALUE 

8 DEFVALUE 

9 LOACCEPT 

10 HI ACCEPT 

11 DIRECTON 

12 DEFWEIGT 



Numeric 
Numeric 
Nvuneric 
Numeric 
Numeric 



2 
2 
2 
2 
2 



Character 
Nvimeric 



** Total ** 



151 



Each detail is stored in the DETAIL file under a unique four character 
code. The detail is described by giving it both a title and a short description. 
The user can indicate whether that detail is an actual value (A) or an estimated 
(E) value for that detail in the KIND field, while the TYPE field is used to 
remind the user whether the detail is a number (N), dollar value (X>), or 
percent (P). These two fields are for memo purposes only, since the system 
does not enforce any special editing on particular values. 

The CAROC/P Decision Support System has been programmed to only 
accept numeric values for details. While this does not always fit every 
situation, most values associated with a detail can be converted into a niuneric 
equivalent for analysis. MINVALUE, MAXVALUE, and DEFVALUE are 
entered into the detail definition file to aid the editing process during actual 
value entry. MINVALUE represents the smallest numeric value fJlowed for 
this particular detail, while MAXVALUE is the largest allowed. DEFVALUE, 
a value that must fall between the two, is the default value assigned to that 
detail for all sections unless otherwise overridden by the user. Proper setting 
of these parameters insures that out-of-range values are not accidentally 
entered into the VALUE data file. 

HIACCEPT, LOACCEPT, and DIRECTON aU deal with the way the 
value is interpreted at analysis reporting time. HIACCEPT typically represents 
the highest acceptable value for the "middle" range, while LOACCEPT 
represents the lowest acceptable value. A section with an actual value for this 
detail greater than HIACCEPT is deemed to be a "positive" ("+"), while one 
with an actual value less than that of LOACCEPT is deemed to be a "negative" 
("-"). The above is true when DIRECTON is set to a value of ">". Sometimes, 
however, "positive" is represented by lesser values, not greater ones. Changing 
DIRECTON to "<" reverses the interpretation of HIACCEPT and LOACCEPT 
so that an actual value below LOACCEPT is rated as "positive" while one 
greater than HIACCEPT is rated as "negative". The user is responsible for 
determining adequate ranges for both LOACCEPT and HIACCEPT, as well as 
the direction of interpretation (DIRECTON). Both LOACCEPT and 
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HIACCEPT must be within the ranges of MINVALUE and MAXVALUE, and 
HIACCEPT must be equal to or greater than LOACCEPT. 

The final field in the DETAIL data file is DEFWEIGT, representing the 
default weight the user typically assigns to this detail at analysis reporting 
time Like KIND and TYPE, this is a memo field, serving only to remind the 
user of this detail's intended use in an analysis. The DETAIL data file contains 
only a single index. Stored under the key of DETAILD, the detail code is used 
primarily for data file maintenance procedures. It is also accessed when 
analysis reports are defined. 

After one (or more) details have been established the user next typically 
enters values for each section under each detail While it is not absolutely 
necessary for every section to have a value for each detail (the system can 
function with that level of missing data), the CAROC/P Decision Support 
System will attempt to insure that each has at least the default. It does this 
by scanning the VALUE file each time the value maintenance program is 
invoked, insuring that every section has a corresponding detail entry for e\cry 
detail. If a section-detail does not previously exist the system will 
automatically create one, using the detail's default value (DEFVALXJE) as the 
value for that section-detail. 

Structur* for database: VALUE. DBF 

Field Field Name Type Width Dec 



1 DETAIL Character 4 

2 COURSE Numeric 8 

3 SECTION Numeric 4 

4 VALUE Numeric 12 2 
*• Total ** 29 



The combination of the DETAIL, and character representations of the 
COURSE and SECTION form a sixteen character string making each entry of 
detail for each course-section unique. Each of these entries contains a single 
numeric value, entered according to the specifications and constraints in the 
DETAIL file for that detail code. 

The VALUE file contains two standing indices: VALUEC and VALUEN. 
Both use the same elements for the keys (given above), though the first is 
sensitive to detail-course-section order while the second is sensitive to course- 
section-detail order. By selecting these keys the user can alternate the 
presentation of records firom this file, dining the data entry procedure, to better 
accommodate the order of the data in its raw form. 

After detail values have been entered for each section the user is ready 
to proceed to the creation of an anal3rsis report. The first step in this process 
is to create one (or more) entries in the ANALYS data file. This file contains 
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the maximmn (up to fifteen) number of details allowed on a particular report, 
naming each of the ptrticular details. In addition, each detail is given a 
weighting to be used at analysis time. Default PROGRAM and LOCATION 
codes are also specified (with the keyword of "ALL" if all of either PROGRAM 
or LOCATION is desired). While the user retains the fi:eedom to readjust the 
weights, program, location, and several other parameters at reporting time a 
particular detail can only be "disabled" by setting its weight to zero. 



StructUTtt for databases AHALYS.DBF 



Fiald 


Fiald Hama 


Typa 


Width 


1 


ANALYS 


Character 


4 


2 


TITLE 


Character 


30 


3 


DESCRIP 


Character 


60 


4 


DETAILl 


Character 


4 


5 


DETAIL2 


Character 


4 


6 


DETAIL3 


Character 


4 


7 


DETAIL4 


Character 


4 


8 


DETAIL5 


Character 


4 


9 


DETAILS 


Character 


4 


10 


DETAIL7 


Character 


4 


11 


DETAILS 


Character 


4 


12 


DETAIL9 


Character 


4 


13 


DETAILl 0 


Character 


4 


14 


DETAILl 1 


Character 


4 


15 


DETAILl 2 


Character 


4 


16 


DETAILl 3 


Character 


4 


17 


DETAILl 4 


Character 


4 


18 


DETAILl 5 


Character 


4 


19 


WEIGHTl 


Nuneric 


3 


20 


WEIGHT2 


Numeric 


3 


21 


WEIGHT3 


Numeric 


3 


22 


WEIGHT4 


Numeric 


3 


23 


WEIGHT5 


Numeric 


3 


24 


WEIGHTS 


Numeric 


3 


25 


WEIGHT? 


Numeric 


3 


26 


WEIGHTS 


Numeric 


3 


27 


WEIGHT9 


Numeric 


3 


28 


WEIGHT 10 


Nvuneric 


3 


29 


WEIGHT 11 


Ntimeric 


3 


30 


WEIGHT 12 


Niuneric 


3 


31 


WEIGHT 13 


Numeric 


3 


32 


WEIGHT14 


Numeric 


3 


33 


WEIGHT 15 


Niimeric 


3 


34 


PROGRAM 


Character 


4 


35 


LOCATION 


Character 


4 


** Total *• 




208 



The final file used in the CAROC/P DSS is the REPDAT data file. This 
data file does not contain actual data entered by the user. Rather, it is a 
temporary file used by the analysis reporting procedure during the process of 
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report creation. NormaUy this ffle is empty, only containing records durmg tiie 
actual report creation. Essentially, it is used as an interim storage me for 
correctly ordering the section records after each's total weighted detail code has 
been computed. 



structure for databaaa: REPDAT^DBP 
Field Field Name Type 



1 

2 
3 
4 
5 
6 
7 
8 
9 



COURSE 
COURSETIT 
SECTION 



12 
13 



16 
17 
18 
19 
20 
21 
22 



3:' 

34 
35 
36 
37 



Nxuneric 
Character 
Numeric 



SECTIONTIT Character 



PROG 
LOC 

DISPLYl 
DISPLY2 
DISPLYS 

10 DISPLYA 

11 DISPLY5 
DISPLY6 
DISPLY7 

14 DISPLY8 

15 DISPLY9 
DISPLYIO 
DISPLYll 
DISPLY12 
DISPLY13 
DISPLY14 
DISPLY15 
TOTALWGT 

23 VALWGTl 

24 VALWGT2 

25 VALWGT3 

26 VALWGT4 

27 VALWGT5 

28 VALWGT6 

29 VALWGT7 

30 VALWGT8 

31 VALWGT9 

32 VALWGTIO 
VALWGTll 
VALWGT12 
VALWGT13 
VALWGT14 
VALWGT15 



Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Numeric 

N\imeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 



Total ** 



Width 

8 
30 
4 

30 
4 
4 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
3 
12 
12 
12 
12 
12 
12 
12 
12 
12 
12 
12 
12 
12 
12 
12 
279 



Dec 



2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 



The DISPLY fields store the interpretation of the value for each of the 
fifteen details (either "+ "o", or "-"), while the VALWGT fields store the actual 
values. At report printing time the values are multipUed by their percentage 
of their weight (adjusted according to the total of all of the weights for that 
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reoort). Interpretations of a earn a relative score of 200 a o a score of 
S^d k V' a score of 0. Factored with each detaU's relative weight pves 
each^ction a total rank score in the range of 0 to 200. regardless of the 
number of details included in the report. 

In order for the analysis report to print ^^^'^.^.pS^^^^^^ 
order the REPDAT file is indexed on a smgle tag named WEHjHlb. ims tag 
is a fifteen character combination of the rank order value, the coi^ae number, 
and the section number. In this way different sections tied with the same 
relative ranking come out sorted in course-section order on the analysis report. 



The Prog raTn Files 

Fifteen different program files were written for the CAROC/P Decision 
Support System, encompassing over 5.000 Unes of Dbase IV progr^ code. 
These program files are divided into four distinct groups: Menu and Control, 
Data File Maintenance, Data File Reporting, and Analysis Reportmg. The 
system was designed so that each of the fifteen modules could be run 
independently of the others in a stand-alone environment. When compiled a^d 
bound, the entire program becomes a single, executable file named ROCF- 
DSS.DBO, which can be executed firom either the Dbase IV executive or the 
Dbase IV Rxmtime. 

The Menu and Control program files consist of the ROCP-DSS.PRG and 
MAINMENU.PRG files. These two files provide the main and sub-menus of 
the DSS, including the specifications necessary for linking between the 
different program modules and their respective data files. 

Each of the five data files is associated with its own independent 
maintenance program: MANALYS.PRG, MCOURSE.PRG. MDETAIL.PRG, 
MSECTION.PRG. and MVALUE.PRG. The full operation of these programs 
are described in the system's user manual. EssentiaUy, each module is 
intended to allow the user to add, modify, or tag for deletion records 
appropriate to that data file. The exception to that is Jie MVALUE.PRG 
routine, which automatically adds records to the VALUE data file when 
necessary (the user is prohibited firom manually adding records to this file). 

Two other routines also exist to support file maintenance. 
PACKALL.PRG is a routine that removes all tagged for delete records firom 
each of the five data files, reindexing each in the process. UPDATE.PRG is a 
stand-alone module that must be accessed outside of the regular menu ^stem. 
This procedure can be used if, for some unknown reasons, the standing indices 
of any of the files should become corrupt or othtrwise out of sync with the 
regular data files. This most typically occurs when a user adds records 
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manually to the data files, such a might occur when data is imported from an 
existing records maintenance ^stem. UPDATE.PRG can then be run to 
rebuild all of the indices necessaxy for the correct operation of the DSS. 

Five reporting procedures also exist to create basic reports of each of 
main data files. These procedures are: RANALYS.FRG, RCOURSE.FRG, 
KDETAIL.FRG, RSECTION.FRG, and RVALUE.FRG. These routines are 
standard Dbase IV reporting routines, allowing the user to dump the entire 
contents of each of the data files in a standard report format. 

The most compUcated procedure of the Decision Support System is the 
RREPORT.PRG procedure. This procedure actually produces the analysis 
report, combining the data previoudy entered into each of the five data files. 
This routine first queries the user for the analysis code they want to use for 
the given report, previously entered into the ANALYS data file. Once that 
code's parameters have been retrieved the user can make manual adjustments 
to the LOACCEPT, HIACCEPT, DIRECTON, and WEIGHT values of each 
detail specified (up to the maximum of the fifteen allowed). The user can also 
specify particular programs or locations to restrict the report to. Once the 
report specifications have been made this routine extracts the necessary data 
firom the five data files, writing the required fields to the REPDAT data file. 
Finally, the REPDAT data file is read back in according to the index order and 
the printed report produced 
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Document 9 



Illustrated Benefits of the ROCP-DSS software 
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Benefits of 
the MIS Model 

• Integrates information 

• Clarifies priorities 

• Establishes standards 

• Interprets available data 

• Refines data definitions 

• Strengthens communication 

• Monitors accounting/reporting 

• Insures local control 
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Integrates Information 



ROCP-DSS 




Attendance 




Finance 




Inventory 



Student 
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Establishes Local Standards 



Poor 


Expected 


Good 




0 


\ 




1 » 


J 1 



0% 25% 50% 75% 100% 

Distribution of Section Performance 
on a Typical Quality Indicator 
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Clarifies Priorities 



Should we 
retain this course? 



O O o 
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Interprets Available Data 



Can the labor market 
support this course? 



o 




Placement 
trends 



Student 
interest 
trends 



New jobs 
available 



Projected 
market growth 
rate 



Separations 
from job market 




Advisory 
input on 
curriculum 
relevance 



Numbers 
similarly trained 
elsewhere 



Other training 
agencies in 
the region 
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Refines Data Definitions 



What is quality 
instruction ? 




'mm- 



Quality 
of the 



facility 



Placements 



Teacher 
qualifications 



Teacher 
experience 



Linkages with 
business and industry 




Active 

advisory 

group 



Equipment 
quality and 
availability 



Completions 



^ Evaluation by 
instructional 
supervisors 



Student feedback 



Accuracy and timeliness 
of reporting attendance* 
certifications, and grades. 
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strengthens Communication 




Students 
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Monitors Accounting and Reporting 




Electronic 
transmission of 
required 
report data 




Hard copy for 
local distribution 
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Insures Local Control 




Regulatory 
Control 
through 

Indicators 




Management 
Control through 
Documentation 
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